One of the requirements of a large scale web application is the ability to utilize load balancers to control user access to the application. A key component of successful high availability is to ensure that users can transfer from one IAP server to another without having to login again. To avoid this, IAP must store all login tokens in a shared location. To facilitate this, our IAP High Availability system utilizes a Redis/Sentinel solution in a Master/Slave configuration to keep tokens available to all IAP servers. This solution also provides failover capabilities between IAP servers in the event a single Redis server should fail.
Note: The following is a living document of information that is continually revised and updated. As such, it may not complement the IAP Installation Guide precisely. We recommend that you check back for additional updates and if any questions, please contact Itential Customer Success for additional support.
Official Redis Documentation
Please consult the Redis documentation for a full description on installation, configuration, and operation of Redis/Sentinel configuration.
Our recommendation for Shared Token Redis is to use three (3) Redis and three (3) Sentinels. The number of servers recommended is three (3) per data center with a single Redis/Sentinel on each server. This provides the best combination of failover and support. The Redis/Sentinel servers can be co-located on other servers such as RabbitMQ or IAP, but this is not a highly recommended practice as it can impact performance and maintenance. We also do not recommend locating Redis/Sentinel on the Mongo servers due to Mongo high usage of disk performance and memory usage.
Example of Basic Master Config
bind <current server ip> 127.0.0.1 requirepass "<SecretPassword>" masterauth "<SecretPassword>"
Example of Basic Slave Config
bind <current server ip> 127.0.0.1 requirepass "<SecretPassword>" masterauth "<SecretPassword>" slaveof <master server ip> 6379
Example of Basic Sentinel Config
bind <current server ip> 127.0.0.1 port 26379 sentinel monitor <redis cluster name> <master redis server ip> 6379 2 sentinel down-after-milliseconds <redis cluster name> 10000 sentinel failover-timeout <redis cluster name> 30000 sentinel auth-pass <redis cluster name> "<SecretPassword>"
IAP Specific Configuration
To manage failover Itential recommends the following settings be set on the Redis/Sentinel configuration. The will lower the failover times between Redis servers when the Sentinel detects a Redis is offline. This is done to ensure no delay in the user experience while the failover takes place. The provided values are recommended based on internal test cases.
sentinel down-after-milliseconds mymaster 10000 sentinel failover-timeout mymaster 30000
Security characteristics and recommendations for a Shared Token Redis configuration are outlined as follows.
Security - Encryption
Out of the box IAP to Redis communications lack the functionality to enable SSL encryption due to a current limitation of the Redis application. Redis is currently working to resolve this issue, and details on this can be found here: Redis - Encryption. Because of this limitation, Itential strongly recommends that Shared Token Redis servers have all network access blocked from all outside access other than from IAP servers. The IAP Token should be considered sensitive and confidential information.
In situations with diaster recovery or a second data center, Itential recommends separate Redis/Sentinel setups each connected to the IAP servers in their data center. This means that IAP servers will only share security tokens with the IAP servers in their data center. Sharing tokens between data centers is not recommended due to the lack of encryption and performance.
Security - Encryption Alternatives
When SSL encryptions are required, it is possible to configure a VPN tunnel between IAP and Redis utilizing the encrypted VPN for traffic. Another alternative is to use stunnel to setup encrypted communication between IAP and Redis. These configuration solutions are outside the scope of this document.
Security - Passwords
In addition to network security, Itential recommends that passwords be enabled on Redis servers to help limit access to the token data. The Redis servers, once configured to use a password, will require the password before any access is granted to the tokens stored internally. Please note, Redis does not support user authentication but a single master password for access. For this reason the password should be stored securely and periodically changed.