Introduction¶

moxi is a proxy for memcached traffic that can be used in a variety of ways and is especially useful against a Couchbase Server cluster.

There a three main reasons for using moxi :

Compatibility

You want the automatic, dynamic reconfiguration features of Couchbase, but you’re too busy to change your web application. moxi can help here by proxying requests to the correct Couchbase server nodes and will provide protocol transformations, too. That is, you can keep on using ascii memcached clients without any changes.
Simplification

Simplifying your configuration. moxi can dynamically respond to cluster health updates, so your application doesn’t have to.
Performance

You can use moxi within a “client-side” architecture (which runs on your web application servers). This can be helpful to share already-connected sockets, reduce the number of connections from web-app servers to Couchbase, and avoid extra machine hops.

Using moxi¶

moxi is an integrated part of Couchbase already, so just point your memcached clients to the right port (such as 11211) and you’re done. This common configuration is called server-side moxi.

As mentioned above, you can also run your own moxi instances on the client-side, or on your web-app server hosts. In this case, your web application will be making local machine connections to a long-running moxi process (which will provide connection pooling for you). That is, your web applications will connection to 127.0.0.1:11211, and moxi will do the rest to proxy requests to the right Couchbase server.

For more information on running moxi in client-side, see Standalone Moxi Component

moxi is a proxy server that speaks the memcached protocol, can multiplex multiple clients onto fewer, shared connections, and can translate between memcached ascii and binary protocols. moxi has advanced features that allow it to have dynamic reconfigurability and integration with Couchbase.

Server-side moxi¶

Each Couchbase server node includes integrated (or server-side) moxi server software. Server-side moxi processes are spawned and managed by the Couchbase cluster manager (so they’re running on the same server-side nodes as the other Couchbase server processes). Couchbase knows how to do handle watch-dogging and reconfigures these server-side moxi processes automatically.

There is one “gateway” moxi server process in particular which the Couchbase cluster manager will configure to listen, by default, on port 11211. This gateway moxi server process is fully cluster aware and will be automatically, dynamically reconfigured by the Couchbase cluster manager whenever the cluster changes server node membership or health status.

The gateway moxi has an additional special responsibility of handling the “default” bucket, if there is one. (There will be a default bucket configured when you follow the normal Couchbase initial configuration wizard screens). So a client that connects to port 11211 will end up using the “default” bucket (if there is one).

The gateway moxi also has a special responsibility to handle any buckets that need SASL authentication. That is, a client can use the binary memcached protocol to connect to the gateway moxi and use SASL authentication (which is part of the binary memcached protocol) to associate the client’s connection with a different bucket.

Also, as you create per-port buckets in your Couchbase cluster, the Couchbase cluster manager will automatically spawn and manage a separate moxi process or instance for every port-based bucket that you’ve created. For example, if you create a second bucket named “shoppingCarts” on port 11212, there will be a dedicated moxi process/instance that listens on port 11212 that provides access to that shoppingCarts bucket.

Proxying existing memcached clients¶

Existing ascii memcached clients (what most folks are using) should be able to connect to any gateway moxi instance or to any port-based moxi instance, without any changes. In your application’s client library configuration, just specify the right port (eg, 11211, 11212, etc).

If an ascii memcached client connects to the gateway moxi, it will be assigned to use the “default” bucket (if the default bucket exists (it may have been deleted or not have been created by the Couchbase administrator)).

If an ascii memcached client connects to a port-based moxi (for example, port 11212 for the “shoppingCarts” bucket), it will be assigned to the bucket that was created with that port.

Client side moxi¶

You may also choose to run moxi on the “client-side”. That means installing and running moxi, for example, on your web-application server hosts.

This has the advantage of increased performance, especially for PHP-style applications that use occasional process restart strategies. (Think of classic CGI apps in the worst case.) Since a client-side moxi server can run on the same machine as the memcached client application (usually a web host), the memcached client only needs to make a fast local machine network connection to moxi, the client-side moxi server can reuse long-running connections (saving on TCP connection setup handshake time), and the client-side moxi can multiplex multiple clients into shared connections (reducing the file descriptor usage on the servers.)

Additionally, client-side moxi ’s can provide some simplified configuration benefits. The client applications can connect to the client-side moxi, which might be running at “127.0.0.1:11211” instead of having the manage their own server-lists and having to have their own code to handle cluster reconfigurations. The client-side moxi can participate in the same dynamic cluster reconfiguration algorithms of Couchbase. Any cluster membership or health changes can propagate automatically to the client-side moxi ’s instances.

See Standalone Moxi Component for more information about configuring a client-side moxi.

Data Channel versus Management Channel¶

There are two kinds of connections that moxi either creates or handles: data channel traffic (for memcached key-value requests) and management channel traffic (to learn about the cluster configuration and re-configuration).

Management channel traffic is REST-based, and by default uses port 8091. These are specified via the REST URL’s described in the previous section. The JSON messages passed over these HTTP/REST channels are all about the cluster’s configuration (and re-configuration events).

Data channel traffics occurs on bucket-oriented ports, like 11211. The messages passing over these connections are key-value requests such as get or incr.

Much less traffic passes over the management HTTP/REST channels, as cluster health and membership changes tend to be infrequent.

However, the management channels are important, as a client-side moxi works best when it receives relevant cluster management change information quickly. moxi servers will still work even with slow management channel news, but will inefficiently be making requests to the wrong servers and receiving errors that force moxi to retry requests on other servers.

As mentioned in the previous section, you may apply industry standard tools like HTTP reverse-proxies to help increase availability and ease of configurability. An HTTP proxy can help make a Couchbase proxy perform better because it will improve the management channel (HTTP/REST) operations. By adding a level-of-indirection via a reverse-proxy, you can allow each client-side moxi to have a simple URL as shown above.

Following A Request Through Moxi¶

To understand some of the configurable command-line flags in moxi ( concurrency, downstream_max, downstream_conn_max, downstream_timeout, wait_queue_timeout, etc), it can be helpful to follow a request through moxi…

The normal flow of data for moxi is as follows:

A client connects

A client creates a connection (an upstream conn) to moxi. moxi’s -c command-line parameter ultimately controls the limits on the maximum number of connections.

In this -c parameter, moxi inherits the same behavior as memcached, and will stop accept()‘ing client connections until existing connections are closed. When the count of existing connections drops below the -c defined level, moxi will accept() more client connections.
The client makes a request, which goes on the wait queue

Next, the client makes a request — such as simple single-key command (like set, add, append, or a single-key get).

At this point, moxi places the upstream conn onto the tail of a wait queue. moxi’s wait_queue_timeout parameter controls how long an upstream conn should stay on the wait queue before moxi times it out and responds to the client with a SERVER_ERROR response.
The concurrency parameter

Next, there’s a configurable max limit to how many upstream conn requests moxi will process concurrently off the head of the wait queue. This configurable limit is called concurrency. (This formerly used to be known, perhaps confusingly, as downstream_max. For backwards compatibility, concurrency and downstream_max configuration flags are treated as synonyms.)

The concurrency configuration is per-thread and per-bucket. That is, the moxi process-level concurrency is actually concurrency X num-worker-threads X num-buckets.

The default concurrency configuration value is 1024. This means moxi will concurrently process 1024 upstream connection requests from the head of the wait queue. (There are more queues in moxi, however, before moxi actually forwards a request. This is discussed in later sections.)

Taking the concurrency value of 1024 as an example, if you have 4 worker threads (the default, controlled by moxi’s -t parameter) and 1 bucket (what most folks start out with, such as the “default” bucket), you’ll have a limit of 1024 x 4 x 1 or 4096 concurrently processed client requests in that single moxi process.

The rationale behind the concurrency increase to 1024 for moxi’s configuration (it used to be much lower) is due to the evolving design of moxi. Originally, moxi only had the wait queue as its only internal queue. As more, later-stage queues were added during moxi’s history, we found that getting requests off the wait queue sooner and onto the later stage queues was a better approach. We’ll discuss these later-stage queues below.

Next, let’s discuss how client requests are matched to downstream connections.
Key hashing

The concurrently processed client requests (taken from the head of the wait queue) now need to be matched up with downstream connections to the Couchbase server. If the client’s request comes with a key (like a SET, DELETE, ADD, INCR, single-key GET), the request’s key is hashed to find the right downstream server “host:port:bucket” info. For example, something like — “memcache1:11211:default”. If the client’s request was a broadcast-style command (like FLUSH_ALL, or a multi-key GET), moxi knows the downstream connections that it needs to acquire.
The downstream conn pool

Next, there’s a lookup using those host:port:bucket identifiers into a downstream conn pool in order to acquire or reserve the appropriate downstream conns. There’s a downstream conn pool per thread. Each downstream conn pool is just a hashmap keyed by host:port:bucket with hash values of a linked-list of available downstream conns. The max length of any downstream conn linked list is controlled by moxi’s downstream_conn_max configuration parameter.
- The downstream_conn_max parameter
  
  By default the downstream_conn_max value is 4. A value of 0 means no limit.
  
  So, if you’ve set downstream_conn_max of 4, have 4 worker threads, and have 1 bucket, you should see moxi create a maximum of 4 X 4 X 1 or 16 connections to any Couchbase server.
Connecting to a downstream server

If there isn’t a downstream conn available, and the downstream_conn_max wasn’t reached, moxi creates a downstream conn as needed by doing a connect() and SASL auth as needed.
- The connect_timeout and auth_timeout parameters
  
  The connect() and SASL auth have their own configurable timeout parameters, called connect_timeout and auth_timeout, and these are in milliseconds. The default value for connect_timeout is 400 milliseconds, and the auth_timeout default is 100 milliseconds.
- The downstream conn queue
  
  If downstream_conn_max is reached, then the request must wait until a downstream conn becomes available; the request therefore is placed on a per-thread, per- host:port:bucket queue, which is called a downstream conn queue. As downstream conns are released back into the downstream conn pool, they will be assigned to any requests that are waiting on the downstream conn queue.
- The downstream_conn_queue_timeout parameter
  
  There is another configurable timeout, downstream_conn_queue_timeout, that defines how long a request should stay on the downstream conn queue in milliseconds before timing out. By default, the downstream_conn_queue_timeout is 200 milliseconds. A value of 0 indicates no timeout.
A downstream connection is reserved

Finally, at this point, downstream conn’s are matched up for the client’s request. If you’ve configured moxi to track timing histogram statistics, moxi will now get the official start time of the request. moxi now starts asynchronously sending request message bytes to the downstream conn and asynchronously awaits responses.

To turn on timing histogram statistics, use the “time_stats=1” configuration flag. By default, time_stats is 0 or off.
The downstream_timeout parameter

Next, if you’ve configured a downstream_timeout, moxi starts a timer for the request where moxi can limit the time it will spend processing a request at this point. If the timer fires, moxi will return a “SERVER_ERROR proxy downstream timeout” back to the client.

The downstream_timeout default value is 5000 milliseconds. If moxi sees this time elapse, it will close any downstream connections that were assigned to the request. Due to this simple behavior of closing downstream connections on timeout, having a very short downstream_timeout is not recommended. This will help avoid repetitive connection creation, timeout, closing and reconnecting. On an overloaded cluster, you may want to increase downstream_timeout so that moxi does not constantly attempt to time out downstream connections on an already overloaded cluster, or by creating even more new connections when servers are already trying to process requests on old, closed connections. If you see your servers greatly spiking, you should consider making this adjustment.
Responses are received

When all responses are received from the downstream servers for a request (or the downstream conn had an error), moxi asynchronously sends those responses to the client’s upstream conn. If you’ve configured moxi to track timing histogram statistics, moxi now tracks the official end time of the request. The downstream conn is now released back to the per-thread downstream conn pool, and another waiting client request (if any) is taken off the downstream conn queue and assigned to use that downstream conn.

Backoff/BlacklistingAt step 6, there’s a case where a connect() attempt might fail. Moxi can be configured to count up the number of connect() failures for a downstream server, and will also track the time of the last failing connect() attempt.

With the connect() failure counting, moxi can be configured to blacklist a server if too many connect() failures are seen, which is defined by the connect_max_errors configuration parameter. When more than connect_max_errors number of connect() failures are seen, moxi can be configured to temporarily stop making connect() attempts to that server (or backoff) for a configured amount of time. The backoff time is defined via the connect_retry_interval configuration, in milliseconds.

The default for connect_max_errors is 5 and the connect_retry_interval is 30000 millisecods, that is, 30 seconds.

If you use connect_max_errors parameter, it should be set greater than the downstream_conn_max configuration parameter.

Standalone Moxi Component¶

Couchbase provides a standalone Moxi component that enables scalable deployments across multiple web application servers. The best practice is to install one standalone Moxi per web application server, and then point web applications to the standalone Moxi instance on its own server. The Moxi component then communicates directly with the Cluster Manager (specified via the HTTP/REST URL supplied upon Moxi startup). This Moxi component communicates with the Cluster Manager and is aware of cluster settings and changes.

Downloading and Installing Standalone Moxi¶

To obtain the standalone Moxi component, download it (from http://www.couchbase.com/downloads-all ). Installer packages are available

RedHat Linux

shell> sudo rpm -i moxi-server_VERSION.rpm

Ubuntu

shell> sudo dpkg -i moxi-server_VERSION.deb

Moxi will install into the /opt/moxi/subdirectory, so that you can run:

shell> /opt/moxi/bin/moxi

To get some initial usage/help information:

shell> /opt/moxi/bin/moxi -h

Running moxi¶

You can start and stop moxi from the command line by using the moxi-server startup script:

shell> /etc/init.d/moxi-server start|stop|restart

You can start, stop and restart moxi using the corresponding keyword to the command line.

Configuration¶

In 1.7, Moxi will install as a service and the instructions immediately below are valid. For anything prior to 1.7, please follow the instructions to run Moxi manually

The standalone moxi server includes /etc/init.d/moxi-server startup scripts that you need to configure before using the startup scripts. After installing the moxi server software, please edit the configuration files under the /opt/moxi/etc directory

/opt/moxi/etc/moxi.cfg

Local config/information.
/opt/moxi/etc/moxi-cluster.cfg

Cluster config/information.

Local configuration (moxi.cfg)¶

The moxi.cfg file allows you to configure the moxi process, including process performance characteristics like timeouts, resource limits, listening port number, etc. It is equivalent to what you can specify with moxi’s -Z command-line parameter.

An example of moxi.cfg file:

usr=Administrator,
pwd=password,
port_listen=11211,
default_bucket_name=default,
downstream_max=1024,
downstream_conn_max=4,
downstream_conn_queue_timeout=200,
downstream_timeout=5000,
wait_queue_timeout=200,
connect_max_errors=5,
connect_retry_interval=30000,
connect_timeout=400,
auth_timeout=100,
cycle=200

Cluster Configuration (moxi-cluster.cfg)¶

The moxi-cluster.cfg file allows you to point moxi to a cluster. It is equivalent to moxi’s -z command-line parameter.

It will usually contain the REST URL information of a Couchbase cluster. For example:

url=http://membase01:8091/pools/default/bucketsStreaming/default

Advanced Standalone moxi Configuration¶

Some of you may want to run multiple instances of moxi, giving each moxi process special command-line configuration parameters. The simplest way to start moxi completely via the command-line is by passing it one or more comma-separated REST URL’s:

shell> moxi URL1[,URL2[,URLn]]

For example:

shell> moxi http://membase0:8091/pools/default/bucketsStreaming/shoppingCarts,http://membase1:8091/pools/default/bucketsStreaming/shoppingCarts

The URLs in this case are separated by a comma. In a configuration file, the URLs should be separated by the pipe ( | ) character.

On startup moxi will contact each URL in the provided order, until one of the URL’s succeeds in returning a valid REST response. The connection that moxi creates to the URL is persistent; neither Couchbase server nor moxi will close the HTTP connection. Therefore Couchbase server can stream cluster topology update information to moxi as needed. If moxi loses its persistent HTTP connection to a URL, it will attempt to find a running Couchbase server using the URL’s provided in the start command. In other words, moxi will first attempt to reconnect to Couchbase server using URL’s provided at the beginning of the command, and if needed, then proceed to subsequent URLs.

A per-bucket URL has the form of:

http://<MEMBASE_SERVER>:8091/pools/default/bucketsStreaming/<BUCKET_NAME>

For example, with a bucket named “shoppingCarts”:

shell> moxi http://membase1:8091/pools/default/bucketsStreaming/shoppingCarts

And using more than one Couchbase server for redundancy:

shell> moxi http://membase0:8091/pools/default/bucketsStreaming/shoppingCarts,http://membase1:8091/pools/default/bucketsStreaming/shoppingCarts

The “Gold Standard” best practice (if you have the flexibility) is to put the Couchbase host URLs behind a http reverse-proxy, so that all the client-side Moxi’s can be more easily configured with a “stable” URL which can handle cluster changes without having to touch each client-side Moxi like..

shell> moxi http://membase_http_reverse_proxy_host:8091/pools/default/bucketsStreaming/<BUCKET_NAME>

Certain HTTP reverse proxies or load balancers will close a connection that they have deemed idle. When a client-side Moxi is connected to Couchbase’s streaming API, there is very little data flowing unless topology changes occur. Because of this, some load balancers will close the connection which can cause traffic disruptions. It is a best practice to configure the load balancer with an infinite timeout on idle connections for this traffic.

Additionally, some HTTP reverse proxies or load balancers can provide ‘round robin’ load balancing; such features should be avoided. Because clients connect to the HTTP reverse-proxy/load-balancer, the reverse- proxy/load-balancer in ‘round-robin’ mode will choose the next server on its list to forward the connection. Instead the HTTP reverse-proxy/load-balancer should be configured to use either a ‘source’ mode (different reverse-proxies/load- balancers have different terms for this concept), where the same HTTP client from a given IP address would be proxy for the same HTTP server.

Changing the default listening portMoxi will listen on port 11211 by default. To change Moxi’s listen port, use a -Z port_listen flag:

shell> moxi -Z port_listen=11311 http://membase1:8091/pools/default/bucketsStreaming/shoppingCarts

Specifying SASL authenticationIf your bucket has SASL authentication credentials, you can also specify them using the usr and pwd -Z flags. For example:

shell> moxi -Z usr=shoppingCarts,pwd=need_a_better_pswd,port_listen=11311 \
    http://membase1:8091/pools/default/bucketsStreaming/shoppingCarts

Increased LoggingTo have moxi emit more logging information, you can specify -v, -vv, or -vvv flags (the more v’s, the more verbose moxi will be). For example:

shell> moxi -vv -Z usr=shoppingCarts,pwd=need_a_better_pswd,port_listen=11311  \
    http://membase1:8091/pools/default/bucketsStreaming/shoppingCarts

Ascii and Binary Protocols¶

Moxi, by default, will auto-detect whether your client is using ascii memcached protocol or binary memcached protocol. You can configure moxi to only allow a particular protocol via its “-B” command-line flag. This is the same “-B” command-line feature as memcached.

Gateway versus bucket-specific moxi’s¶

When moxi is running embedded in the Couchbase cluster, it actually is used into two different modes: as a “gateway” moxi or as a bucket-specific moxi. This is done by just starting moxi processes with different command-line flags, specifically by pointing moxi at (slightly) different URL’s. You can use these different URL’s when starting your own client-side, standalone moxi’s.

The gateway moxi (usually listening on port 11211 when moxi is embedded in Couchbase) handles all SASL authenticated buckets. That is, a client connects to port 11211, and by default, ends up on the default bucket (if it exists and hasn’t been deleted). The client can then use binary memcached protocol to SASL authenticate to a different bucket.

The bucket-specific moxi, on the other hand, is already associated with a particular bucket (and this is a bucket that’s not already handled by the gateway moxi). For example, a bucket-specific moxi might be listening on port 11212, for the “shoppingCarts” bucket, and as soon as a client connects to port 11212, the client may immediately use any memcached commands (get/set/delete/flush_all/etc) to affect key-value items in that shoppingCarts bucket.

Gateway moxiTo start a gateway moxi, use a URL of the form: /pools/default/saslBucketsStreaming

For example:

./moxi -Z usr=Administrator,pwd=DontTell http://membase-a:8091/pools/default/saslBucketsStreaming

Per-Bucket moxiTo start a per-bucket moxi, use a URL of the form: /pools/default/bucketsStreaming/<BUCKET_NAME> For example:

./moxi http://membase-a:8091/pools/default/bucketsStreaming/shoppingCarts

Increasing Concurrency¶

There are a couple ways to increase concurrency in moxi:

Increase Worker ThreadsBy default, moxi uses 4 worker threads. This can be changed via the -t THREAD_NUM flag. For example:

shell> moxi -t 6 -Z port_listen=11311 http://membase1:8091/pools/default/bucketsStreaming/shoppingCarts

Increase Number of Concurrent Client Requestsmoxi limits the number of inflight or concurrent client requests that it will send to the Couchbase cluster. This is controlled via the -Z concurrency configuration value, whose default value is 1024. For older users of moxi, the “concurrency” configuration parameter is also known as the “downstream_max” configuration parameter. For example, to set concurrency to 8:

shell> moxi -Z port_listen=11311,concurrency=8 http://membase1:8091/pools/default/bucketsStreaming/shoppingCarts

The formula of total number concurrent requests that moxi will process is:

NUM_THREADS x NUM_BUCKETS x concurrency

With the last command-line, then, the NUM_THREADS defaults to 4. There’s only one bucket involved (the shoppingCarts bucket), so NUM_BUCKETS is 1. And, concurrency has been overridden to be 8. So, there will be moxi allow only 32 concurrent client requests to be processed. All other client requests will go onto a wait queue until active requests are finished.

ImplicationsIncreasing concurrency and the number of worker threads isn’t necessarily a free lunch. More threads, for example, means using resources that other processes (such as your web app servers) might need. Higher concurrency settings from moxi (when multiplied by the number moxi’s you’ve deployed) means more connections will be created to downstream servers, which can eventually cause other performance issues or hit connection (file descriptor) limits.

Downstream conn queuesMore information on the downstream conn queues is available: Following A Request Through Moxi.

Timeouts¶

You can configure a wait queue timeout in moxi, so that moxi will return a SERVER_ERROR if a client request has been waiting too long in the wait queue. You can also specify a timeout for any active, inflight requests (in the previous section example, for example, this would be one of the 32 active requests).

These timeouts are -Z wait_queue_time and downstream_timeout configuration values, specified in milliseconds. To use timeouts, you must specify a clock cycle or quantum, in milliseconds, via the “cycle” -Z flag. The -Z flag is 200 milliseconds by default so that moxi doesn’t waste any effort making timing system calls. In other words, moxi only checks the system time by default every 200 milliseconds. Other threads use the cached value for the system time. This means that other timeout values should be a multiple of the cycle value. For example:

shell> moxi -Z port_listen=11311,concurrency=8,cycle=100,wait_queue_timeout=5000,downstream_timeout=5000 \
     http://membase1:8091/pools/default/bucketsStreaming/shoppingCarts

Please see Following A Request Through Moxi for more information on timeouts.

Configuration File¶

The -Z flag can only be specified once on the moxi command-line, so it can get really long with key-value configuration pairs. Here’s a long example:

shell> moxi -Z port_listen=11311,concurrency=8,cycle=100,wait_queue_timeout=5000,downstream_timeout=5000 \
    http://membase1:8091/pools/default/bucketsStreaming/shoppingCarts

At this point, you can put the entire -Z key-value configuration list into a separate file, and tell moxi to load the -Z configuration from that file. For example:

shell> moxi -Z ./relative/path/to/config_file http://membase1:8091/pools/default/bucketsStreaming/shoppingCarts
shell> moxi -Z /absolute/path/to/config_file http://membase1:8091/pools/default/bucketsStreaming/shoppingCarts

The contents of the config file still needs to be key=value pairs and comma-separated, but may have extra whitespace and line breaks for readability. Here’s an example -Z config file:

port_listen = 11311, concurrency = 8, cycle = 100, wait_queue_timeout = 5000, downstream_timeout = 5000,

The REST/HTTP URL’s may also be placed into their own configuration file. It turns out the way we’ve been specifying REST/HTTP URL’s so far in this document is actually just a convenience shorthand for the following (lowercase) -z url flag. We can the -z flag a cluster configuration flag. For example:

shell> moxi -z url=http://membase1:8091/pools/default/bucketsStreaming

When specifying multiple hosts, use a comma to separate each URL:

shell> moxi -z url=http://membase1:8091/pools/default/bucketsStreaming

The above is exactly the same as the more convenient:

shell> moxi http://membase1:8091/pools/default/bucketsStreaming

However, the -z can also point to a separate file:

shell> moxi -z ./relative/path/to/lowercase-z-config-file
shell> moxi -z /absolute/path/to/lowercase-z-config-file

The contents of the “lowercase z config file” would look like:

url = http://membase1:8091/pools/default/bucketsStreaming

Multiple URL’s can be specified in the cluster REST/URL config file, but must be separated by ‘|’ (vertical ‘pipe’ bar) characters (because the comma character is already used as a key=value delimiter):

url = http://membase1:8091/pools/default/bucketsStreaming|http://membase2:8091/pools/default/bucketsStreaming|http://membase3:8091/pools/default/bucketsStreaming

The URLs in this case are separated by the pipe ( | ) character. On the command-line, they should be separated by a comma.

Finally, you can use both capital Z and lowercase z flags, to specify both kinds of config files:

shell> moxi -Z ./moxi.cfg -z ./cluster.cfg

Errors¶

Command-line and Configuration ErrorsIf you’re pointing moxi at the wrong server/URL, or moxi can’t reach any of its REST/URL’s, you’ll see output prefixed with the word ERROR: :

shell> ./dev/moxi/moxi http://foo.bar.baz.foo
ERROR: could not contact REST server(s): http://foo.bar.baz.foo
ERROR: could not contact REST server(s): http://foo.bar.baz.foo

You can see that moxi retries its configured REST/URL’s every second. Also, if the server that moxi is contacting is up, but provides mangled JSON information, moxi will also inform you every second.

shell> ./dev/moxi/moxi http://google.com
2010-11-01 16:14:43: (agent_config.c.453) ERROR: could not parse JSON from REST server: <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"> <TITLE>301 Moved</TITLE></HEAD><BODY> <H1>301 Moved</H1> The document has moved <A HREF="http://www.google.com/">here</A>. </BODY></HTML>

When you provide moxi with a list of REST/URL’s, the earlier URL’s have higher precedence. So, moxi will favor the first URL and attempt to contact it first. Then the 2nd URL, then the 3rd and so on. If the 2nd URL works, but stops working (moxi succeeded in contacting the 2nd URL, but loses its connection to the 2nd URL some time later), moxi will start over from the start of the REST/URL’s list with the first URL (in the hopes that the favored earlier URL is now back online).

Protocol ErrorsPlease see: Moxi Error Responses

Moxi with Memcached¶

Moxi speaks the memcached protocol, so it can work with both memcached servers and Couchbase servers.

When used with Couchbase, Moxi is started with a REST/URL that describes the Couchbase cluster configuration, and Moxi uses vbucket hashing to determine which key-value items live on which servers.

When used with standard, classic memcached, Moxi should be started with different command-line parameters that will make Moxi use libmemcached (ketama) hashing. In particular, Moxi will take a different cluster or -z command-line flag, like…

./moxi -z LISTEN_PORT=MEMCACHED_HOST1[:PORT1][,MEMCACHED_HOSTN[:PORTN]]

The memcached PORT’s default to 11211. For example, below, moxi will listen on port 11811 and use ketama hashing against a server-list of mc1:11211,mc2:11211 :

./moxi -z 11811=mc1,mc2
              ./moxi -z 11811=mc1:11211,mc2:11211

You may also move the -z cluster configuration into a separate file, by specifying either an absolute or relative path…

./moxi -z ./relative/path/to/cluster/config_file
              ./moxi -z /absolute/path/to/cluster/config_file

The contents of the config_file should look like…

11811=mc1:11211,mc2:11211

Moxi Performance Tuning Standalone Moxi¶

As your site and cluster usage grows, it’s often a good step to move towards standalone or client-side Moxi. (It can be even better, too, to move towards a “smart client”, which removes the need for moxi (whether server-side or client-side) in the first place.)

See Standalone Moxi Component for more information.

Connection Counts¶

For standalone (client-side) moxi, a key resource to watch is connection usage, especially when bumping into O/S limits (eg, number of file descriptors).

There are two sides of this, client-side (upstream) and Couchbase server-side (downstream).

Upstream ConnectionsIn general, moxi usually has no trouble servicing many hundreds or more of connections from upstream clients (eg, from your application). If you need to tune this, use the -c command-line parameter (see moxi -h for more moxi command-line usage information).

However, moxi doesn’t let all client/upstream connections attack the servers all at once, but has a (per-thread) connection pool and a configurable concurrency limit on the number of upstream/client requests it will send to downstream servers.

See the “concurrency” (aka, “downstream_max”) configuration parameter, as discussed here: [Standalone Moxi Component](#moxi-standalone) andFollow A Request Through Moxi

Downstream ConnectionsThe other side of moxi, of course, is with its Couchbase server-facing connections. Usually, you won’t run into limitations here, but should instead consider the equation from Couchbase server’s point of view.

You might have NUM_MOXI number of standalone moxi processes all feverishly processing requests at their downstream_max limits of concurrency. And, you might have NUM_MEMBASE number of Couchbase nodes.

So, from each standalone moxi, you’ll have these many connections to a single Couchbase node:

NUM_CONN_FROM_MOXI_TO_MEMBASE = NUM_MOXI_WORKER_THREADS x NUM_BUCKETS x downstream_max

To simplify, let’s say you just have a single bucket (such as the “default” bucket), that downstream_max is 4 (the default, circa 1.6.0 moxi) and that NUM_MOXI_WORKER_THREADS is 4 (which is also the default -t command-line flag value, circa 1.6.0 moxi). So, in this example, there are 4 x 1 x 4 = 16 connections that a single moxi will make to Couchbase.

Multiply NUM_CONN_FROM_MOXI_TO_MEMBASE by NUM_MOXI and you’ll have the number of connections that each Couchbase will see from the farm of standalone moxi processes. (In your accounting, don’t forget to leave some room for other clients (such as Couchbase’s own monitoring processes, vbucketmigrator processes and embedded moxi’s so that they can create their own connections to Couchbase services).

Multiply NUM_CONN_FROM_MOXI_TO_MEMBASE by NUM_MEMBASE and you’ll have the number of connections that each moxi will make just on its downstream side.

These numbers (plus whatever whatever extra wiggle room you need) can help you calculate your ulimit’s correctly.

Current ConnectionsTo see the current number of connections into moxi, as totals and on the upstream and/or downstream-side, send moxi an ascii “stats proxy” request. More information on this is at: Moxi Statistics

Operating System Tuning¶

Some networking stack settings to consider tuning for your deployments:

Turning on tcp_no_delay in your application code
Setting the MTU on the localhost to 1500 (linux defaults can be much higher)

Moxi Statistics¶

Cluster-wide stats vs Single-server stats

moxi provides a “cluster-wide” view of stats. That is, moxi broadcasts the “stats” command to every server in the cluster and aggregates (sums) the returned values that make sense, before returning aggregated values back to you. Underneath the hood, it uses a hashmap to do the aggregation, so that stats values that moxi emits are returned in un-sorted (confusing) order. Because of this, folks usually do something like the following to make values easier to read (port 11211 is where moxi is listening by default when used as part of the Couchbase installation).

echo "stats" | nc HOST 11211 | sort

If you just want to get a single server’s stats, directly from memcached/ep-engine, please use:

/opt/membase/VERSION/bin/ep-engine/management/stats.py HOST:11210 all

Port 11210 is where memcached/ep-engine is running by default in Couchbase.

The latest versions of moxi have improved proxy-related stats:

telnet 127.0.0.1 11211
stats proxy

Alternatively:

echo "stats proxy" | nc HOST 11211

The above will provide statistics about the moxi proxy itself, not from asking the Couchbase servers for cluster-wide statistics.

The proxy-related stats can tell you information about the moxi process, including:

how it’s been configured
how many buckets it’s handling
per-bucket statistics
number of connections, requests, errors, retries, etc.

The stats proxy output will looks like:

$ echo stats proxy | nc 127.0.0.1 11211
STAT basic:version 1.6.1rc1
STAT basic:nthreads 5
STAT basic:hostname stevenmb.local
STAT memcached:settings:maxbytes 67108864
STAT memcached:settings:maxconns 1024
STAT memcached:settings:tcpport 0
....
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_upstream_paused 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_upstream_unpaused 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:err_oom 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:err_upstream_write_prep 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:err_downstream_write_prep 0 END

Let’s walk through each section.

Basic Information¶

Here, we can tell that moxi has been configured with 4 worker threads plus the main listener thread. 4 + 1 == 5 “nthreads”

STAT basic:version 1.6.1rc1
STAT basic:nthreads 5
STAT basic:hostname stevenmb.local

Memcached Inherited Settings And Statistics¶

Next, moxi emits many statistics and settings that it inherits from its memcached codebase. (moxi was based on the memcached codebase, inheriting a lot of asynchronous, high-performance C-based connection and networking machinery.)

Of note, the “memcached:settings” are changable through moxi’s command-line flags. You can use “moxi -h” to learn more about moxi’s command-line flags.

In particular, the “maxconns” (-c command-line flag) may be a limit that your moxi might reach, if you have many clients connecting to the same moxi and/or have a large cluster (many Couchbase servers). Look for the “listen_disabled_num” (in a later section) to grow rapidly in this situation.

And, we can see that moxi’s been running only a short time (“uptime” of 11 seconds), is running the 64-bit version (“pointer_size”), and does not have many open connections (“curr_connections”, which counts both client connections into moxi and also moxi’s connections to downstream Couchbase servers).

STAT memcached:settings:maxbytes 67108864
STAT memcached:settings:maxconns 1024
STAT memcached:settings:tcpport 0
STAT memcached:settings:udpport -2
STAT memcached:settings:inter NULL
STAT memcached:settings:verbosity 0
STAT memcached:settings:oldest 0
STAT memcached:settings:evictions on
STAT memcached:settings:domain_socket NULL
STAT memcached:settings:umask 700
STAT memcached:settings:growth_factor 1.25
STAT memcached:settings:chunk_size 48
STAT memcached:settings:num_threads 5
STAT memcached:settings:stat_key_prefix :
STAT memcached:settings:detail_enabled no
STAT memcached:settings:reqs_per_event 20
STAT memcached:settings:cas_enabled yes
STAT memcached:settings:tcp_backlog 1024
STAT memcached:settings:binding_protocol auto-negotiate
STAT memcached:stats:pid 691
STAT memcached:stats:uptime 11
STAT memcached:stats:time 1288649197
STAT memcached:stats:version 1.6.1rc1
STAT memcached:stats:pointer_size 64
STAT memcached:stats:rusage_user 0.015575
STAT memcached:stats:rusage_system 0.005112
STAT memcached:stats:curr_connections 4
STAT memcached:stats:total_connections 5
STAT memcached:stats:connection_structures 5
STAT memcached:stats:cmd_get 0
STAT memcached:stats:cmd_set 0
STAT memcached:stats:cmd_flush 0
STAT memcached:stats:get_hits 0
STAT memcached:stats:get_misses 0
STAT memcached:stats:delete_misses 0
STAT memcached:stats:delete_hits 0
STAT memcached:stats:incr_misses 0
STAT memcached:stats:incr_hits 0
STAT memcached:stats:decr_misses 0
STAT memcached:stats:decr_hits 0
STAT memcached:stats:cas_misses 0
STAT memcached:stats:cas_hits 0
STAT memcached:stats:cas_badval 0
STAT memcached:stats:bytes_read 24
STAT memcached:stats:bytes_written 0
STAT memcached:stats:limit_maxbytes 67108864
STAT memcached:stats:accepting_conns 1
STAT memcached:stats:listen_disabled_num 0
STAT memcached:stats:threads 5
STAT memcached:stats:conn_yields 0

Moxi-Specific Global Settings¶

Next, moxi emits many moxi-specific configuration values. The word “behavior” is a synonym here for configuration (and turns out to be an easy target for grep'ing). Of note, we can see that moxi concurrency setting (“downstream_max”) is the default value of 4, and timeouts have been configured (by default) to off (“cycle”, “wait_queue_timeout”, and “downstream_timeout” are 0).

Additionally, the front_cache and key_stats features are configured off (the default), via the “front_cache_lifespan” and “key_stats_lifespace” of 0. And, moxi has received two dynamic reconfiguration events so far (“stat_configs” is 2).

STAT proxy_main:conf_type dynamic
STAT proxy_main:behavior:cycle 0
STAT proxy_main:behavior:downstream_max 4
STAT proxy_main:behavior:downstream_conn_max 0
STAT proxy_main:behavior:downstream_weight 0
STAT proxy_main:behavior:downstream_retry 1
STAT proxy_main:behavior:downstream_protocol 8
STAT proxy_main:behavior:downstream_timeout 0
STAT proxy_main:behavior:wait_queue_timeout 0
STAT proxy_main:behavior:time_stats 0
STAT proxy_main:behavior:connect_max_errors 0
STAT proxy_main:behavior:connect_retry_interval 0
STAT proxy_main:behavior:front_cache_max 200
STAT proxy_main:behavior:front_cache_lifespan 0
STAT proxy_main:behavior:front_cache_spec
STAT proxy_main:behavior:front_cache_unspec
STAT proxy_main:behavior:key_stats_max 4000
STAT proxy_main:behavior:key_stats_lifespan 0
STAT proxy_main:behavior:key_stats_spec
STAT proxy_main:behavior:key_stats_unspec
STAT proxy_main:behavior:optimize_set
STAT proxy_main:behavior:usr Administrator
STAT proxy_main:behavior:host
STAT proxy_main:behavior:port 0
STAT proxy_main:behavior:bucket
STAT proxy_main:behavior:port_listen 11211
STAT proxy_main:behavior:default_bucket_name default
STAT proxy_main:stats:stat_configs 2
STAT proxy_main:stats:stat_config_fails 0
STAT proxy_main:stats:stat_proxy_starts 2
STAT proxy_main:stats:stat_proxy_start_fails 0
STAT proxy_main:stats:stat_proxy_existings 1
STAT proxy_main:stats:stat_proxy_shutdowns 0

Per-Bucket Statistics¶

Next, we see moxi emits settings and statistics about its buckets. The first bucket in this example is the “default” bucket, available on port 11211.

We can see that the “default” bucket in moxi inherits the same behavior settings from the global configuration settings (the same “behavior” values as above).

STAT 11211:default:info:port 11211
STAT 11211:default:info:name default
STAT 11211:default:info:config { "name": "default", "nodeLocator": "vbucket", "saslPassword": "", "
nodes": [{ "clusterMembership": "active", "status": "healthy"}],
STAT 11211:default:info:config_ver 2
STAT 11211:default:info:behaviors_num 1
STAT 11211:default:behavior:downstream_max 4
STAT 11211:default:behavior:downstream_conn_max 0
STAT 11211:default:behavior:downstream_weight 0
STAT 11211:default:behavior:downstream_retry 1
STAT 11211:default:behavior:downstream_protocol 8
STAT 11211:default:behavior:downstream_timeout 0
STAT 11211:default:behavior:wait_queue_timeout 0
STAT 11211:default:behavior:time_stats 0
STAT 11211:default:behavior:connect_max_errors 0
STAT 11211:default:behavior:connect_retry_interval 0
STAT 11211:default:behavior:front_cache_max 200
STAT 11211:default:behavior:front_cache_lifespan 0
STAT 11211:default:behavior:front_cache_spec
STAT 11211:default:behavior:front_cache_unspec
STAT 11211:default:behavior:key_stats_max 4000
STAT 11211:default:behavior:key_stats_lifespan 0
STAT 11211:default:behavior:key_stats_spec
STAT 11211:default:behavior:key_stats_unspec
STAT 11211:default:behavior:optimize_set
STAT 11211:default:behavior:usr default
STAT 11211:default:behavior:host
STAT 11211:default:behavior:port 0
STAT 11211:default:behavior:bucket
STAT 11211:default:behavior:port_listen 11211
STAT 11211:default:behavior:default_bucket_name default

Per-Bucket Server-List Settings¶

Next, in keys that follow a naming pattern of PORT:BUCKET_NAME:behavior-X, we can see the “server list” that moxi associates with a bucket. We can see that moxi knows of only 1 server (X == 0) for the “default” bucket:

STAT 11211:default:behavior-0:downstream_weight 0
STAT 11211:default:behavior-0:downstream_retry 1
STAT 11211:default:behavior-0:downstream_protocol 8
STAT 11211:default:behavior-0:downstream_timeout 0
STAT 11211:default:behavior-0:usr default
STAT 11211:default:behavior-0:host 127.0.0.1
STAT 11211:default:behavior-0:port 11210
STAT 11211:default:behavior-0:bucket

Per-Bucket Statistics (Counters)¶

Next, we can see the active counters that moxi tracks on a per-bucket basis. First, we can see that nothing’s happened to the frontcache counters (unsurprisingly, as the frontcache feature has been configured off):

STAT 11211:default:stats:listening 2
STAT 11211:default:stats:listening_failed 0
STAT 11211:default:frontcache:max 0
STAT 11211:default:frontcache:oldest_live 0
STAT 11211:default:frontcache:tot_get_hits 0
STAT 11211:default:frontcache:tot_get_expires 0
STAT 11211:default:frontcache:tot_get_misses 0
STAT 11211:default:frontcache:tot_get_bytes 0
STAT 11211:default:frontcache:tot_adds 0
STAT 11211:default:frontcache:tot_add_skips 0
STAT 11211:default:frontcache:tot_add_fails 0
STAT 11211:default:frontcache:tot_add_bytes 0
STAT 11211:default:frontcache:tot_deletes 0
STAT 11211:default:frontcache:tot_evictions 0

Then, we can see more interesting and useful per-bucket statistics.

In general, counters named num_xxx are active counters (they increase and decrease). And, tot_xxx are counters that only increase (until there’s a “stats proxy reset” command).

Below, “upstream” refers to client-side statistics (from your application). And “downstream” refers to Couchbase server-side statistics.

Of note, moxi acquires downstream connections from a connection pool, assigns them to upstream clients, and releases the downstream connections back into the connection when the upstream client has finished a request/response, if there were no errors (in which case moxi might close the downstream connection rather than releasing it back into the connection pool). Because of this, you’ll see that “tot_downstream_conn_acquired” will often grow (hopefully, only slightly) faster than “tot_downstream_conn_released”.

STAT 11211:default:pstd_stats:num_upstream 1
STAT 11211:default:pstd_stats:tot_upstream 1
STAT 11211:default:pstd_stats:num_downstream_conn 0
STAT 11211:default:pstd_stats:tot_downstream_conn 0
STAT 11211:default:pstd_stats:tot_downstream_conn_acquired 0
STAT 11211:default:pstd_stats:tot_downstream_conn_released 0
STAT 11211:default:pstd_stats:tot_downstream_released 0
STAT 11211:default:pstd_stats:tot_downstream_reserved 0
STAT 11211:default:pstd_stats:tot_downstream_reserved_time 0
STAT 11211:default:pstd_stats:max_downstream_reserved_time 0
STAT 11211:default:pstd_stats:tot_downstream_freed 0
STAT 11211:default:pstd_stats:tot_downstream_quit_server 0
STAT 11211:default:pstd_stats:tot_downstream_max_reached 0
STAT 11211:default:pstd_stats:tot_downstream_create_failed 0
STAT 11211:default:pstd_stats:tot_downstream_connect 0
STAT 11211:default:pstd_stats:tot_downstream_connect_failed 0
STAT 11211:default:pstd_stats:tot_downstream_connect_timeout 0
STAT 11211:default:pstd_stats:tot_downstream_connect_interval 0
STAT 11211:default:pstd_stats:tot_downstream_connect_max_reached 0
STAT 11211:default:pstd_stats:tot_downstream_waiting_errors 0
STAT 11211:default:pstd_stats:tot_downstream_auth 0
STAT 11211:default:pstd_stats:tot_downstream_auth_failed 0
STAT 11211:default:pstd_stats:tot_downstream_bucket 0
STAT 11211:default:pstd_stats:tot_downstream_bucket_failed 0
STAT 11211:default:pstd_stats:tot_downstream_propagate_failed 0
STAT 11211:default:pstd_stats:tot_downstream_close_on_upstream_close 0
STAT 11211:default:pstd_stats:tot_downstream_timeout 0
STAT 11211:default:pstd_stats:tot_wait_queue_timeout 0
STAT 11211:default:pstd_stats:tot_assign_downstream 0
STAT 11211:default:pstd_stats:tot_assign_upstream 0
STAT 11211:default:pstd_stats:tot_assign_recursion 0
STAT 11211:default:pstd_stats:tot_reset_upstream_avail 0
STAT 11211:default:pstd_stats:tot_multiget_keys 0
STAT 11211:default:pstd_stats:tot_multiget_keys_dedupe 0
STAT 11211:default:pstd_stats:tot_multiget_bytes_dedupe 0
STAT 11211:default:pstd_stats:tot_optimize_sets 0
STAT 11211:default:pstd_stats:tot_optimize_self 0
STAT 11211:default:pstd_stats:tot_retry 0
STAT 11211:default:pstd_stats:tot_retry_time 0
STAT 11211:default:pstd_stats:max_retry_time 0
STAT 11211:default:pstd_stats:tot_retry_vbucket 0
STAT 11211:default:pstd_stats:tot_upstream_paused 0
STAT 11211:default:pstd_stats:tot_upstream_unpaused 0
STAT 11211:default:pstd_stats:err_oom 0
STAT 11211:default:pstd_stats:err_upstream_write_prep 0
STAT 11211:default:pstd_stats:err_downstream_write_prep 0

Of note in the previous section, if “tot_upstream_paused” grows much greater than “tot_upstream_unpaused”, that means that client connections are stuck in a “paused” state, awaiting some event that wakes them up (such as moxi receiving a response from a Couchbase server or a reserved/assigned downstream connection getting closed).

NULL_BUCKET Statistics¶

The NULL_BUCKET also has its own section of emitted moxi statistics. If you see a NULL_BUCKET in your “stats proxy” output, it’s because you’re talking to a moxi that’s been started (via command-line arguments) as a “gateway moxi”. In this situation, when a client first connects, it is assigned to this “/dev/null” bucket. The client then can do a SASL authentication to be associated with a non-null bucket.

Timing Histograms¶

If you start moxi with the -Z time_stats=1 flag:

./moxi -Z time_stats=1,port_listen=11511 http://HOST:8080/pools/default/bucketsStreaming/default

Then you can get histogram output from moxi:

telnet 127.0.0.1 11511
stats proxy timings

When time_stats=1, moxi will call gettimeofday() system call before and after each request. And, per-bucket, it introduces into moxi a relatively small constant amount of memory to track the histogram bins. By default time_stats is 0 (or disabled).

For example:

$ echo "stats proxy timings" | nc 127.0.0.1 11511
STAT 11511:default:connect 0+100=1 25.00% ********
STAT 11511:default:connect 100+100=3 100.00% ************************
STAT 11511:default:connect 200+100=0 100.00%
STAT 11511:default:reserved 0+100 =97624 86.18% ************************
STAT 11511:default:reserved 100+100 =15414 99.79% ***
STAT 11511:default:reserved 200+100 =134 99.90%
STAT 11511:default:reserved 300+100 =29 99.93%
STAT 11511:default:reserved 400+100 =21 99.95%
STAT 11511:default:reserved 500+100 =10 99.96%
STAT 11511:default:reserved 600+100 =14 99.97%
STAT 11511:default:reserved 700+100 =10 99.98%
STAT 11511:default:reserved 800+100 =9 99.99%
STAT 11511:default:reserved 900+100 =3 99.99%
STAT 11511:default:reserved 1000+100 =2 99.99%
STAT 11511:default:reserved 1100+100 =3 99.99%
STAT 11511:default:reserved 1200+100 =1 99.99%
STAT 11511:default:reserved 1300+100 =1 99.99%
STAT 11511:default:reserved 1400+100 =0 99.99%
STAT 11511:default:reserved 1500+100 =0 99.99%
STAT 11511:default:reserved 1600+100 =0 99.99%
STAT 11511:default:reserved 1700+100 =0 99.99%
STAT 11511:default:reserved 1800+100 =0 99.99%
STAT 11511:default:reserved 1900+100 =0 99.99%
STAT 11511:default:reserved 2000+100 =0 99.99%
STAT 11511:default:reserved 2100+200 =0 99.99%
STAT 11511:default:reserved 2300+400 =0 99.99%
STAT 11511:default:reserved 2700+800 =0 99.99%
STAT 11511:default:reserved 3500+1600=4 100.00%
STAT 11511:default:reserved 5100+3200=2 100.00%
STAT 11511:default:reserved 8300+6400=0 100.00%
END

- There are two parts to the histogram:
- connect – time spent in a connect
reserved – time spent with a downstream fully reserved (or assigned) to an upstream connection. To read it, a line looks like:
STAT…:reserved bucket_start_usec+bucket_width_usec =count cummulative_percentile asterisk_graph For example:

STAT 11511:default:reserved 0+100 =97624 86.18% ************************
STAT 11511:default:reserved 100+100 =15414 99.79% ***
STAT 11511:default:reserved 200+100 =134 99.90%

And that means that:

97,624 requests fell into the 0-to-100 usec bucket, which was 86.18% of all requests.

15,414 requests fell into the 100-to-200 usec bucket, and 99.79% of requests were faster than 200 usecs.

134 requests fell into the 200-to-300 usec bucket, and 99.90% of requests were faster than 300 usecs.

Note that the histogram bucket (or bin) widths start out constant; but, the bucket widths start doubling towards the end with the longer timings.

Proxy Configuration¶

Recent builds of moxi (> 1.6.1) support a “stats proxy config” command, which can show you the exact configuration REST/JSON that moxi is using.

For example, if moxi was started with ketama hashing:

$ ./dev/moxi/moxi -z 11211=127.0.0.1:11411,127.0.0.1:11511

Then “stats proxy config” would return:

$ echo stats proxy config | nc 127.0.0.1 11211
STAT 11211:default:config 127.0.0.1:11411,127.0.0.1:11511
END

If moxi was started with Coucbase REST/URL/JSON, then you’ll see JSON:

$ echo stats proxy config | nc 127.0.0.1 11211
STAT 11211:default:config {
"name": "default",
"nodeLocator": "vbucket",
"saslPassword": "",
...clipped...
}
STAT 11211:[ <NULL_BUCKET> ]:config
END

Resetting Moxi Proxy Stats¶

Moxi’s stats proxy counters are all kept in moxi memory only. Restarting moxi reset those stats. To dynamically reset moxi’s stats proxy counters, send an ASCII request of:

stats proxy reset

Only the stats counters that make sense for reset will be zero'ed – for example, usually the tot_XXXX counters that normally only climb upwards.

For example:

get a
VALUE a 0 1
1
END

stats proxy timings
STAT 11211:default:connect 5100+3200=1 100.00% ************************
STAT 11211:default:connect 8300+6400=0 100.00%
STAT 11211:default:reserved 1200+100 =1  50.00% ************************
STAT 11211:default:reserved 1300+100 =0  50.00%
STAT 11211:default:reserved 1400+100 =0  50.00%
STAT 11211:default:reserved 1500+100 =0  50.00%
STAT 11211:default:reserved 1600+100 =0  50.00%
STAT 11211:default:reserved 1700+100 =0  50.00%
STAT 11211:default:reserved 1800+100 =0  50.00%
STAT 11211:default:reserved 1900+100 =0  50.00%
STAT 11211:default:reserved 2000+100 =0  50.00%
STAT 11211:default:reserved 2100+200 =0  50.00%
STAT 11211:default:reserved 2300+400 =0  50.00%
STAT 11211:default:reserved 2700+800 =0  50.00%
STAT 11211:default:reserved 3500+1600=1 100.00% ************************
STAT 11211:default:reserved 5100+3200=0 100.00%
END

stats proxy reset
OK

stats proxy timings
END

get a
VALUE a 0 1
1
END

stats proxy timings
STAT 11211:default:reserved 300+100=1 100.00% ************************
STAT 11211:default:reserved 400+100=0 100.00%
END

Moxi Error Responses¶

Errors are reported differently based on whether you are using the ASCII or binary protocol.

ASCII protocol error messagesBesides the normal memcached error codes ( CLIENT_ERROR..., SERVER_ERROR..., NOT_FOUND, NOT_STORED ), moxi has its own set of proxy-specific error responses. These follow the SERVER_ERROR prefix used by memcached ascii protocol.

SERVER_ERROR proxy downstream closed

Moxi returns SERVER_ERROR proxy downstream closed when moxi sees a connection it previously had to a downstream Couchbase server has closed.

This can be a transient error response. That is, if Couchbase server just had a very quick, transient connectivity issue, moxi should heal on the next request, where moxi might immediately attempt a re-connect() to handle the next request. The might is due to moxi’s blacklisting/backoff feature. moxi tracks the number of errors it sees during its re-connect attempts. If moxi counts up too many errors during its re-connect attempts, it will blacklist that uncontactable server for awhile. Requests to other online servers will continue as normal. The default configuration values for moxi for connect_max_errors is 5 and for the connect_retry_interval is 30000 (the number of milliseconds that moxi should blacklist a server that reaches connect_max_errors number of consecutive connect() errors).
SERVER_ERROR proxy write to downstream

This is a variation of moxi losing its connection to a downstream Couchbase server, but is returned when moxi sees the error through a different codepath (while trying to write to a socket to a downstream Couchbase server). Pragmatically, SERVER_ERROR proxy write to downstream and SERVER_ERROR proxy downstream closed can be treated as synonyms.
SERVER_ERROR proxy downstream timeout

In this error response, moxi reached a timeout while waiting for a downstream server to respond to a request. That is, moxi did not see any explicit errors such as a connection going down, but the response is just taking too long. The downstream connection will be also closed by moxi rather than putting the downstream connection back into a connection pool. The default downstream_timeout configuration is 5000 (milliseconds).
SERVER_ERROR unauthorized, null bucket

The SERVER_ERROR unauthorized, null bucket response is returned by moxi when your client connection is associated with the NULL BUCKET and the client is trying to perform key-value operations on items in the NULL BUCKET. This usually means your client has not yet done SASL authentication (and the default bucket has been deleted by your Couchbase Administrator).

You may also see this error very early right after bucket creation, when the bucket creation configuration change is still propagating throughout the Couchbase cluster and its clients.
SERVER_ERROR a2b not_my_vbucket

The SERVER_ERROR a2b not_my_vbucket error response is returned by moxi during a Rebalance operation when it cannot find a server that owns the vbucket that owns a given item key-value.

When moxi receives a not-my-vbucket error response from a downstream Couchbase server, moxi will start looking for the right Couchbase server that should own that vbucket. If moxi was provided a fast-forward-map, it will use that information to ask the expected new owner of the vbucket. Otherwise, moxi will probe all the servers in the cluster, serially, and twice to see if it can find the Couchbase server that owns the vbucket. If no Couchbase server owns that vbucket, moxi will finally return the SERVER_ERROR a2b not_my_vbucket error response to your client application.

Binary protocol error messagesMoxi will respond to the client with the same binary error response message that it received from the downstream Couchbase server.

For all other proxy-related errors, moxi will respond with a binary error message with the status/error code of:

PROTOCOL_BINARY_RESPONSE_EBUSY - during a timeout
PROTOCOL_BINARY_RESPONSE_ENOMEM - if moxi or Couchbase server runs out of memory
PROTOCOL_BINARY_RESPONSE_EINTERNAL - if moxi cannot contact a Couchbase server or a connection to a Couchbase server is closed or lost

Moxi Design Internals¶

moxi was started around Q1 2009 based off the memcached code base. It’s current source repository is a http://github.com/couchbase/moxi.

Moxi inherited a lot of machinery from memcached, including:

Networking and connection management
Thread management
Protocol parsing
Startup components
Build environment

From the point of view of moxi’s codebase:

upstream – the client application, or the source of requests
downstream – the Couchbase servers which are the destination for requests

Moxi code also uses the word “pool” in a way that predates our modern usage of “cluster”, so they are somewhat synonyms.

Guiding Ideas¶

Originally, a rule was to keep changes from any “memcached heritage” files to a minimum.

The motivation was so that moxi could keep up with memcached, allowing us to easily merge in any improvements from memcached into moxi.
This is why much of moxi lives in files separate from memcached’s files (cproxy_*.c/.h)
Nowadays, that rule is very much relaxed. We’ll probably never get easy merges from memcached into moxi (or vice versa).

As much as possible, moxi tries to avoid locks, even though it is multi-threaded, especially in the key/item request proxying codepaths.

Moxi can be considered a store and forward proxy. So, the high level “main loop” in moxi worker threads in moxi wait for i/o activity, via libevent (a higher-level library above epoll/poll/kpoll/select). Some of the i/o events include:

If a client has sent a request, a worker thread wakes up, reads in the client’s request fully, and queues each request in memory.
If a downstream cluster resource becomes available, moxi assigns a waiting request off the queue, makes routing decisions (ketama or libvbucket), and writes the request to the appropriate downstream server(s).
If a downstream server has sent a response, a worker thread wakes up, reads in the server’s response fully, and asynchronously writes the response back to the waiting client. After all responses are fully written, the client connection is then available for more input request reading.

Key Dependencies¶

As with memcached, moxi depends on libevent and pthreads.

In addition, moxi has an abstraction layer called mcs where different hashing libraries can be hooked in:

libvbucket
libmemcached

Moxi also depends on the libconflate library, for dynamic, late-binding network based configuration (and re-configuration).

Key Modules / Files¶

To a large degree, moxi’s modules map to the corresponding component files:

agent_* - integration with libconflate for dynamic reconfiguration
mcs - abstraction API over libvbucket and libmemcached
cproxy_config - API for configuration data structures
cproxy_front - the “front caching” feature of moxi
cproxy_multiget - support for multigets and optimizations (de-duplicating keys, de-mux'ing responses, etc)
cproxy_protocol_* - protocol translations
- a - stands for ascii
- b - stands for binary
- the files include:
cproxy_protocol_a - parses and processes ascii client messages
cproxy_protocol_a2a
cproxy_protocol_a2b
cproxy_protocol_b - parses and processes binary client messages
cproxy_protocol_b2b
a2a, a2b, b2b stand for ascii-to-ascii, ascii-to-binary, and binary-to-binary respectively
or, more explicitly a2b stands for ascii-upstream to a binary-downstream

For completeness, here are some other files of lower-level functionality:

inherited from memcached:
- memcached - main(argc,argv), cmd-line parsing, and connection state machine
- thread - multithreading support
- items - API for items
- assoc - slabber aware hashtable
  - slabs - slabber allocator
  - http://code.google.com/p/memcached/wiki/MemcachedSlabAllocator
- cache - utility memory allocator
- genhash - utility hash table
stdin_check - feature to exit() when stdin closes, motivated by windows support.
other utility
- htgram - histogram
- matcher - simple string pattern matcher
- util - other utility functions
- work - multi-threaded work/message queues for communicating between threads
- cJSON - json parsing library

Command-Line Configuration¶

Moxi inherits command-line configuration parsing from memcached (argc/argv/getopt), such as:

-t NUM_THREADS
-vvv
-u USER
-c MAX_CONNS
-d (daemonize)

The new parts of moxi configurability, however, belong with two new flags: -z and -Z.

Specifying a cluster¶

The -z flag specifies a cluster of Couchbase servers. There a several ways to specify a cluster using -z. For example:

``` -z 11211=MC_HOST1:MC_PORT1,MC_HOST2:MC_PORT2

* This is a libmemcached-style of cluster specification, via a list of
  comma-separated HOST:PORT’s.


Here, moxi will listen on port 11211.

``` -z 11211={ some libvbucket JSON config here }
- This is useful for testing libvbucket-style cluster configuration.
- Again, moxi will listen on port 11211.

``` -x url=http://HOST:8080/pools/default/bucketsStreaming/BUCKET_NAME

* This specifies a REST URL where moxi can dynamically request a libvbucket-style
  cluster configuration.

The following, by the way, is an ease-of-use shortcut for specifying a -z url=URL : ./moxi http://HOST:8080/pools/default/bucketsStreaming/BUCKET_NAME

<a id="moxi-internals-cmdline-optional"></a>

### Optional configuration parameters

The -Z flag specifies optional, comma-separated key-value configuration
parameters, called "proxy\_behavior"'s in the codebase. For example:

-Z downstream_max=8,time_stats=1,downstream_protocol=ascii

Here's some of the more useful -Z flags:

port_listen // IL: Default port to listen on. cycle // IL: Clock resolution in millisecs. downstream_max // PL: Downstream concurrency per worker thread. downstream_retry // SL: How many times to retry a cmd, defaults to 1. downstream_protocol // SL: Favored downstream protocol, defaults to binary. downstream_timeout // SL: Millisecs. wait_queue_timeout // PL: Millisecs. time_stats // IL: Capture timing stats (for histograms).

connect_max_errors // IL: Pause when too many connect() errors. connect_retry_interval // IL: Time in millisecs before retrying // when too many connect() errors, to not // overwhelm the downstream servers.

front_cache_max // PL: Max # of front cachable items. front_cache_lifespan // PL: In millisecs. front_cache_spec // PL: Matcher prefixes for front caching. front_cache_unspec // PL: Don’t front cache prefixes.

key_stats_max // PL: Max # of key stats entries. key_stats_lifespan // PL: In millisecs. key_stats_spec // PL: Matcher prefixes for key-level stats. key_stats_unspec // PL: Don’t key stat prefixes.

optimize_set // PL: Matcher prefixes for SET optimization.

usr // SL: User name for REST or SASL auth. pwd // SL: Password for REST or SASL auth. bucket // SL: Bucket name for bucket selection.

default_bucket_name // ML: The named bucket (proxy->name) // that upstream conn’s should start on. // When empty (“”), then only binary SASL // clients can actually do anything useful.

The IL/ML/PL/SL markers above in the -Z flags refer to a behavior inheritance
design in moxi. More information on how behavior/configuration values are
inherited at various levels in the system (initialization-level, pool-level,
server-level...) is available at:

[http://github.com/couchbase/moxi/blob/master/doc/moxi/configuration.org](http://github.com/couchbase/moxi/blob/master/doc/moxi/configuration.org)

<a id="moxi-internals-cmdline-configfiles"></a>

### Config Files

The -z and -Z flags can also be specified via config files, too:

 * -z./relative/path/to/z-config

 * -Z./relative/path/to/Z-config

 * -z /absolute/path/to/z-config

 * -Z /absolute/path/to/Z-config

For improved readability, moxi allows for whitespace in configuration files. For
example, a -Z config file may look like:

cycle = 200, downstream_max = 8, time_stats = 1

<a id="moxi-internals-cmdline-multicluster"></a>

### Multi-cluster configuration

Moxi also supports proxying to multiple clusters from a single moxi instance,
where this was originally designed and implemented for software-as-a-service
purposes. Use a semicolon (';') to specify and delimit more than one cluster:

-z “LISTEN_PORT=[CLUSTER_CONFIG][;LISTEN_PORT2=[CLUSTER_CONFIG2][]]”

For example:

-z “11211=mc1,mc2;11311=mcA,mcB,mcC”

Above, moxi will listen on port 11211, proxying to cluster mc1,mc2. And moxi
will also listen on port 11311, proxying to mcA,mcB,mcC. The multi-pool
codepaths are also overloaded and re-used to support multi-tenancy (multiple
buckets).

<a id="moxi-internals-cmdline-zstored"></a>

### Zstored inherited configuration

The terrific engineers at Zynga provided several enhancements to moxi, with some
cmd-line changes that don't fit into the -Z/-z world:

-O path/to/log/file -X (mcmux compatibility – mcmux was a fork of moxi whose features have been merged back into moxi)

<a id="moxi-internals-threading"></a>

## Threading

If you know memcached threading, moxi's is very much the same.

Just like with memcached, there's a main thread, which is responsible for:

 * main(argc,argv) initialization and command-line parsing.

 * daemonizing.

 * spawning worker and ancilliary threads.

 * listen()'ing on (usually) one or more ports.

 * accept()'ing client connections, and delegating accepted upstream connections to
   worker threads.

The ancillary threads include:

 * libconflate thread - for dynamic (re-)configuration

   This thread is spawned by the libconflate library to make out-of-band REST or
   XMPP calls for configuration instructions.

 * stdin\_term\_handler - watches to see when stdin closes, for windows platform
   support.

 * hashtable reallocation thread - inherited from memcached.

Tere are also worker threads, which are assigned ownership of accepted upstream
connections my the main thread.

<a id="moxi-internals-threads-worker"></a>

### Worker Threads

The number of worker threads is configurable at startup and remains constant
afterwards for the life of the moxi process, based on the -t NUM\_THREADS
command-line parameter.

After a worker thread is assigned to "own" an upstream connection, that upstream
connection is now handled by just that one worker thread. Moxi takes advantage
of this design to allow it to avoid a lot of unnecessary locking, since there's
hardly any communication between worker threads.

<a id="moxi-internals-startup"></a>

## Startup Sequences

Moxi's startup codepaths are slightly different, depending if you have a static
configuration or a dynamic configuration. For example, you have a static
configuration (where moxi has all the complete info it needs to operate) if you
start moxi like:

./moxi -z 11211=memcached1,memcached2,memcached3

You have a dynamic configuration when moxi needs to gets a full configuration
from some remote system, and there might be delays. For example:

./moxi -z url=http://HOST:8080/pools/default/bucketsStreaming/default ```

Static Configuration Startup SequenceWhen moxi knows its listen ports and cluster at start-time:

The main thread parses cmd-line parameters.
The main thread then creates one or more listening sockets.
The main thread then spawns the specified number of worker threads.
The main thread then goes into the libevent main event loop, awaiting connections from clients or other work tasks.

Dynamic Configuration Startup SequenceWhen moxi goes through dynamic configuration:

The main thread parses cmd-line parameters.
The main thread then passes config info, such as URL’s, to libconflate.
The main thread then spawns the specified number of worker threads. (same as above)
The main thread then goes into the libevent main event loop, awaiting connections from clients or other work tasks. (same as above)

Libconflate spawns a separate thread to make REST calls and process REST/JSON configuration responses.

When the libconflate thread receives a proper REST/JSON configuration, it puts a re-configuration work task on the main thread’s work queue.

The work queue functionality is the main way that threads communicate between each other in moxi.

Inter-Thread Communcation & Work Queues¶

The main thread and the worker threads each have their own work queues.

The work queue code is a generalization of code inherited from memcached. The memcached code, which we’ll call the connection assignment queue, is used when the main thread wants to assign a new, accept()‘ed upstream connection to a worker thread. That connection assignment queue code still exists and is used in moxi (it’s not broken). However, the work queue code allows any work task to be assigned to a different thread.

Work TasksA work task is a function pointer plus opaque callback data.

Work Queues & LibeventBoth the connection assignment queue code and the work queue code integrate with libevent, and use the memcached-inspired trick of writing one byte to a pipe, if necessary, in order to wakeup a target thread that might be waiting for file-descriptor activity in libevent.

Asynchronous vs Synchronous Work TasksThe work queue facility is fundamentally asynchronous, so it supports fire-and-forget work tasks.

The work queue code also has functions that support higher-level synchronous work tasks between threads. That is, a calling thread can fire off a work task at some target thread, do some other stuff, then wait (block) for the work task to complete (when the target thread marks the task as complete).

Most communication between the libconflate thread and the main thread uses the synchronous work task facility.

Scatter/Gather Work TasksThe work queue code also has functions that support “broadcast” scatter/gather patterns. For example, the main thread can fire off “statistics gathering” work tasks at each worker thread, and then wait/block until there are responses from all worker threads.

Most communication between the main thread and worker threads uses the scatter/gather facility.

Re-configuration¶

In the dynamic configuration world, configuration, whether the first time or whether a re-configuration, uses the work queues and the same code paths. That is, the first time is just an edge case of re-configuration.

Configuration Lock Avoidance & VersioningAlso, to keep locking to a minimum, each thread has its own copy of the configuration data structures and strings. Each copy of the configuration is associated with a version number (called “config_ver”), to allow for faster comparisons. That is, it’s fast to test that configuration hasn’t changed by just comparing numbers.

In the dynamic configuration world, it all starts with libconflate.

Integration with LibconflateLibconflate has its own, dedicated thread for making HTTP/REST calls to retrieve a cluster configuration and processing JSON responses. When the libconflate receives a proper REST response, it invokes moxi callbacks (so the dynamic re-configuration callbacks are happening on libconflate’s dedicated thread).

Those moxi callbacks are implemented by the agent_config.c file in moxi. The agent_config code next sends a synchronous work task (with the latest, successfully parsed config info to moxi’s main thread.

One JSON message has all bucketsNote that there may be more than one proxy config in a single “re-configuration” work task. The reason is that each JSON configuration message, for simplicity, includes all buckets, even if most of the buckets haven’t changed. For example, if there are 55 buckets, the JSON message will have 55 bucket configuration details in it.

Later, if another bucket is added, for example, the next JSON reconfiguration message will have 56 buckets in it (as opposed to just sending a delta).

This approach is wasteful on network bandwidth (during the infrequent re-configuration episodes), but allows for code simplicity. In this simplicity, the first-time pathway, re-configuration pathway, and approach to handling restarts are all the same, since everything that moxi needs to operate will appear in a single JSON message.

Main thread’s list of proxiesThe main thread tracks a list of active proxies. Each active proxy has a name, such as “default”. In a multi-tenant deployment, this proxy name is the same as a bucket name. During the re-configuration work task, the main thread walks through its active proxies list and updates each proxy data structure

appropriately. Also, new proxies are created and no longer unlisted proxies are deleted, as necessary.

proxy vs proxy_tdWhile the main thread is responsible for proxy data structures (and the linked list of proxy structures), each worker thread has its own copy or snapshot of this information (plus more thread-specific data), called a proxy_td data structure. This is short for “proxy thread data”, and is often abbreviated as “ptd” in the code.

A proxy keeps track of its proxy_td’s (one proxy_td per worker thread).

A worker thread can freely mutate its proxy_td structure, but never touch the proxy_td’s of other work threads. So, a worker thread does not need locks to access its own proxy_td.

A worker thread (and the main thread, too) must use locking before accessing the shared, mutatable fields on its “parent” proxy data structure. Some proxy data structure fields are inherently read-only and static, so they don’t need locking. The code comments (cproxy.h) try to clearly specify which fields are immutable and lockless.

Worker threads handle a proxy_td change.Between each request, each worker thread grabs a very short lock on its parent’s proxy data structure to compare config_ver numbers. If the config_ver numbers match, the worker thread knows that the proxy’s configuration remains the same, and the request can proceed.

If the numbers are different, the worker thread knows a re-configuration has occured, and the worker thread will remove cached information and update its proxy_td copy appropriately.

Appendix: Release Notes¶

The following sections provide release notes for individual release versions of Moxi. To browse or submit new issues, see Moxi Issues Tracker.

Release Notes for Moxi 2.5 GA¶

Fixed Issues in 2.5

MB-9677: A memory leak in the client-side moxi occurs during rebalance.

MB-9549: A memory leak occurs when the TTL is updated for a non-resident item.

MB-8724: Moxi 1.8.1 leaks memory and crashes via the Linux OOM killer.

Release Notes for Moxi 1.8.0 GA (Already released)¶

Fixed Issues in 1.8.1

Known Issues in 1.8.0

When using Moxi in a cluster using haproxy, it’s possible for a memory leak to cause a problem in Moxi when the topology appears to change. The problem is due to haproxy disabling open connections, particularly those used for management, that Moxi may have open, but not using. The haproxy closes these open connections, which moxi identifies as topology changes. The problem is particularly prevalent when using the balance roundrobin load balancing type.

Workaround : There are two possible workarounds to prevent the memory leak in moxi :
- Use balance source load balancing mode within haproxy. This reduces the effect of haproxy closing the open network ports.
- Increase the network timeouts in haproxy. You can do this by editing the haproxy configuration file and adding the following two lines:
```
timeout client 300000
 timeout server 300000
```
  The above sets a 5 minute timeout. You can increase this to a larger number.

Introduction¶

Using moxi¶

Server-side moxi¶

Proxying existing memcached clients¶

Client side moxi¶

Data Channel versus Management Channel¶

Following A Request Through Moxi¶

Standalone Moxi Component¶

Downloading and Installing Standalone Moxi¶

Running moxi¶

Configuration¶

Local configuration (moxi.cfg)¶

Cluster Configuration (moxi-cluster.cfg)¶

Advanced Standalone moxi Configuration¶

Ascii and Binary Protocols¶

Gateway versus bucket-specific moxi’s¶

Increasing Concurrency¶

Timeouts¶

Configuration File¶

Errors¶

Moxi with Memcached¶

Moxi Performance Tuning Standalone Moxi¶

Connection Counts¶

Operating System Tuning¶

Moxi Statistics¶

Proxy-related stats¶

Basic Information¶

Memcached Inherited Settings And Statistics¶

Moxi-Specific Global Settings¶

Per-Bucket Statistics¶

Per-Bucket Server-List Settings¶

Per-Bucket Statistics (Counters)¶

NULL_BUCKET Statistics¶

Timing Histograms¶

Proxy Configuration¶

Resetting Moxi Proxy Stats¶

Moxi Error Responses¶

Moxi Design Internals¶

Guiding Ideas¶

Key Dependencies¶

Key Modules / Files¶

Command-Line Configuration¶

Specifying a cluster¶

Inter-Thread Communcation & Work Queues¶

Re-configuration¶

Appendix: Release Notes¶

Release Notes for Moxi 2.5 GA¶

Release Notes for Moxi 1.8.0 GA (Already released)¶