diff --git a/CHANGELOG.md b/CHANGELOG.md index 015220c5a..97fd5d556 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -405,4 +405,8 @@ Magento CE 1.8+ or EE 1.13+, see [these instructions](https://github.com/nexcess * [#848] Replace Nexcessnet_Turpentine_Model_Dummy_Request with Magento model (@craigcarnell) * [#849] escape | character (@craigcarnell) * Better escaping of double slashes in urls (for better hitrate) (@joolswills) - * [#860] Removing ref to nonexistent file from modman (@cgrice ) + * [#860] Removing ref to nonexistent file from modman (@cgrice ) + * [#596] update docs to make an important notice on Crawler IP Addresses setting under Caching Options (@arosenhagen) + * [#878] Explicit cache bypass for progress sections (@astorm) + * [#844] Fix for HTTPS ESI URLs (@jeroenvermeulen) + * [#865] Implemented load balancing support (@jeroenvermeulen) diff --git a/TECHNICAL_NOTES.md b/TECHNICAL_NOTES.md index bf1b1a434..857bd6ad4 100644 --- a/TECHNICAL_NOTES.md +++ b/TECHNICAL_NOTES.md @@ -39,6 +39,10 @@ Even with the addition of ESI, a client still needs a way to get their session c It sounds simple enough, but Varnish doesn't actually have any easy way to generate unique session tokens. At first, this seemed like a wash but then I remembered that you can actually write straight C code in the VCL for more advanced functionality, such as generating the session token! So that's what Turpentine does: a small C function is included in the VCL to [generate uuids](https://github.com/nexcess/magento-turpentine/blob/master/app/code/community/Nexcessnet/Turpentine/misc/uuid.c), then that function is used to [make a session cookie](https://github.com/nexcess/magento-turpentine/blob/master/app/code/community/Nexcessnet/Turpentine/misc/version-3.vcl#L58) if no session cookie was sent in the request, and the cookie is [passed back to the client](https://github.com/nexcess/magento-turpentine/blob/master/app/code/community/Nexcessnet/Turpentine/misc/version-3.vcl#L340) in the response so it is sent with future requests. +#### Inline C as of Varnish 4.0 + +As of Varnish 4.0 the inline C functionality is disabled by default. To enable inline C you need to modify Varnish's startup config and add `-p vcc_allow_inline_c=on` to Varnish's startup command. + ### Serializing Registry Data To include the registry data required for ESI block rendering in the ESI URL, Turpentine simply takes a list of the needed registry keys (provided in the layout file) for the block and serializes it using PHP's native [`serialize`](http://us3.php.net/serialize) function which turns stores the data in a string which we can then include in the URL as a simple GET parameter. However, a registry key can be associated with any data type, including whole objects. While `serialize` will typically work on objects there are some edge cases (such as XML documents) that cannot be serialized. In order to accommodate these documents, Turpentine considers some objects to be "[complex registry data](https://github.com/nexcess/magento-turpentine/blob/master/app/code/community/Nexcessnet/Turpentine/Model/Observer/Esi.php#L441)" and handles those separately from the standard PHP types like string, int, simple classes, etc. Complex registry data is considered to be Magento "models" that correspond to database records and have `getId` methods (typically things like products, categories, etc). Then Turpentine can simply serialize the model's class and ID, then load the model with the ID instead of serializing the entire object. This neatly sidesteps the whole issue of finding a way to serialize objects that are unserializable. @@ -58,7 +62,7 @@ This change effectively broke Turpentine for those Magento versions, and finding 1. Because the form key is located inside HTML attributes (`href="http://example.com/example/action/form_key//"` for example), by default Varnish doesn't see the ESI include tag which completely breaks the buttons and forms that use the form key. 2. How does Magento know when to generate the actual form key, and not the ESI tag that is replaced by the form key. For example, the form key ESI is needed when generating the "Add to Cart" button, but the actual form key is needed both for both the ESI request to pull in the form key, and when that button is clicked so that it can be compared to the form key that was submitted. -The first is solved by a Varnish config change. Adding `-p esi_syntax=0x2` to Varnish's startup command tells it to look for any ESI tags, even if they're not in properly structured HTML syntax (like the example). The second was more tricky, at first it seemed a static list of actions that use the form key would be needed (like the original fix) that could be used to tell the mage/core session class when to generate the form key, and when to generate the form key ESI tag. However, the solution was staring me in the face: the request itself tells you when you need the actual form key, as it's only needed when the request includes the form key in the GET params or POST data to do the comparison between the actual form key, and the form key sent with the request. Thus we can just [check](https://github.com/nexcess/magento-turpentine/blob/master/app/code/local/Mage/Core/Model/Session.php#L53) if the form key was sent in the request and either generate the real form key, or the ESI tag. +The first is solved by a Varnish config change. Adding `-p esi_syntax=0x2` (`-p feature=+esi_ignore_other_elements` as of Varnish 4.0) to Varnish's startup command tells it to look for any ESI tags, even if they're not in properly structured HTML syntax (like the example). The second was more tricky, at first it seemed a static list of actions that use the form key would be needed (like the original fix) that could be used to tell the mage/core session class when to generate the form key, and when to generate the form key ESI tag. However, the solution was staring me in the face: the request itself tells you when you need the actual form key, as it's only needed when the request includes the form key in the GET params or POST data to do the comparison between the actual form key, and the form key sent with the request. Thus we can just [check](https://github.com/nexcess/magento-turpentine/blob/master/app/code/local/Mage/Core/Model/Session.php#L53) if the form key was sent in the request and either generate the real form key, or the ESI tag. ## Closing Time diff --git a/app/code/community/Nexcessnet/Turpentine/Block/Core/Messages.php b/app/code/community/Nexcessnet/Turpentine/Block/Core/Messages.php index 924a9de06..b09ecd45a 100644 --- a/app/code/community/Nexcessnet/Turpentine/Block/Core/Messages.php +++ b/app/code/community/Nexcessnet/Turpentine/Block/Core/Messages.php @@ -198,16 +198,17 @@ protected function _toHtml() { } else { $this->_loadMessages(); $this->_loadSavedMessages(); - $html = $this->_real_toHtml(); + if ( count($this->getMessages()) ) { + $html = $this->_real_toHtml(); + } else { + // Prevent returning an empty + $html = ''; + } } } else { $html = $this->_real_toHtml(); } $this->_directCall = false; - - if (count($this->getMessages()) == 0) { - return ''; - } return $html; } diff --git a/app/code/community/Nexcessnet/Turpentine/Block/Product/Compared.php b/app/code/community/Nexcessnet/Turpentine/Block/Product/Compared.php new file mode 100644 index 000000000..12ec356c9 --- /dev/null +++ b/app/code/community/Nexcessnet/Turpentine/Block/Product/Compared.php @@ -0,0 +1,34 @@ +getCount()) { + return $this->renderView(); + } + + $this->setRecentlyComparedProducts($this->getItemsCollection()); + + return parent::_toHtml(); + } +} diff --git a/app/code/community/Nexcessnet/Turpentine/Block/Product/Viewed.php b/app/code/community/Nexcessnet/Turpentine/Block/Product/Viewed.php new file mode 100644 index 000000000..9561243e1 --- /dev/null +++ b/app/code/community/Nexcessnet/Turpentine/Block/Product/Viewed.php @@ -0,0 +1,32 @@ +getCount()) { + return $this->renderView(); + } + $this->setRecentlyViewedProducts($this->getItemsCollection()); + return parent::_toHtml(); + } +} diff --git a/app/code/community/Nexcessnet/Turpentine/Helper/Esi.php b/app/code/community/Nexcessnet/Turpentine/Helper/Esi.php index 338ae300a..6e150f44b 100644 --- a/app/code/community/Nexcessnet/Turpentine/Helper/Esi.php +++ b/app/code/community/Nexcessnet/Turpentine/Helper/Esi.php @@ -357,7 +357,10 @@ public function getFormKeyEsiUrl() { $this->getEsiScopeParam() => 'global', $this->getEsiCacheTypeParam() => 'private', ); - return Mage::getUrl( 'turpentine/esi/getFormKey', $urlOptions ); + $esiUrl = Mage::getUrl( 'turpentine/esi/getFormKey', $urlOptions ); + // setting [web/unsecure/base_url] can be https://... but ESI can never be HTTPS + $esiUrl = preg_replace( '|^https://|i', 'http://', $esiUrl ); + return $esiUrl; } /** diff --git a/app/code/community/Nexcessnet/Turpentine/Model/Config/Select/LoadBalancing.php b/app/code/community/Nexcessnet/Turpentine/Model/Config/Select/LoadBalancing.php new file mode 100644 index 000000000..2a891f0e4 --- /dev/null +++ b/app/code/community/Nexcessnet/Turpentine/Model/Config/Select/LoadBalancing.php @@ -0,0 +1,34 @@ +'no', 'label'=>$helper->__('No, use only one backend server')), + array('value'=>'yes', 'label'=>$helper->__('Yes, use load balancing')), + array('value'=>'yes_admin', 'label'=>$helper->__('Yes, with separate settings for Admin')), + ); + } +} diff --git a/app/code/community/Nexcessnet/Turpentine/Model/Config/Select/Version.php b/app/code/community/Nexcessnet/Turpentine/Model/Config/Select/Version.php index 488b26c7d..3bdce45a0 100644 --- a/app/code/community/Nexcessnet/Turpentine/Model/Config/Select/Version.php +++ b/app/code/community/Nexcessnet/Turpentine/Model/Config/Select/Version.php @@ -1,23 +1,23 @@ '2.1', 'label' => $helper->__( '2.1.x' ) ), array( 'value' => '3.0', 'label' => $helper->__( '3.0.x' ) ), + array( 'value' => '4.0', 'label' => $helper->__( '4.0.x' ) ), array( 'value' => 'auto', 'label' => $helper->__( 'Auto' ) ), ); } diff --git a/app/code/community/Nexcessnet/Turpentine/Model/Observer/Esi.php b/app/code/community/Nexcessnet/Turpentine/Model/Observer/Esi.php index b492a9ca5..881bf748d 100644 --- a/app/code/community/Nexcessnet/Turpentine/Model/Observer/Esi.php +++ b/app/code/community/Nexcessnet/Turpentine/Model/Observer/Esi.php @@ -286,6 +286,10 @@ public function injectEsi( $eventObject ) { } $esiUrl = Mage::getUrl( 'turpentine/esi/getBlock', $urlOptions ); + if( $esiOptions[$methodParam] == 'esi' ) { + // setting [web/unsecure/base_url] can be https://... but ESI can never be HTTPS + $esiUrl = preg_replace( '|^https://|i', 'http://', $esiUrl ); + } $blockObject->setEsiUrl( $esiUrl ); // avoid caching the ESI template output to prevent the double-esi- // include/"ESI processing not enabled" bug diff --git a/app/code/community/Nexcessnet/Turpentine/Model/Varnish/Admin.php b/app/code/community/Nexcessnet/Turpentine/Model/Varnish/Admin.php index 0cb5714e8..a26f5105e 100644 --- a/app/code/community/Nexcessnet/Turpentine/Model/Varnish/Admin.php +++ b/app/code/community/Nexcessnet/Turpentine/Model/Varnish/Admin.php @@ -138,22 +138,39 @@ protected function _testEsiSyntaxParam( $socket ) { $result = false; if( $helper->csrfFixupNeeded() ) { - $value = $socket->param_show( 'esi_syntax' ); - if( preg_match( '~(\d)\s+\[bitmap\]~', $value['text'], $match ) ) { - $value = hexdec( $match[1] ); - if( $value & self::MASK_ESI_SYNTAX ) { //bitwise intentional - // setting is correct, all is fine + if ( $socket->getVersion()==='4.0' ) { + $paramName = 'feature'; + $value = $socket->param_show( $paramName ); + $value = explode("\n", $value['text']); + if ( isset($value[1]) && strpos($value[1], '+esi_ignore_other_elements')!==false ) { $result = true; } else { - $session->addWarning( 'Varnish esi_syntax param is ' . + $session->addWarning( 'Varnish feature param is ' . 'not set correctly, please see these instructions ' . 'to fix this warning.' ); } } else { + $paramName = 'esi_syntax'; + $value = $socket->param_show( $paramName ); + if( preg_match( '~(\d)\s+\[bitmap\]~', $value['text'], $match ) ) { + $value = hexdec( $match[1] ); + if( $value & self::MASK_ESI_SYNTAX ) { //bitwise intentional + // setting is correct, all is fine + $result = true; + } else { + $session->addWarning( 'Varnish esi_syntax param is ' . + 'not set correctly, please see these instructions ' . + 'to fix this warning.' ); + } + } + } + + if ( $result===false ) { // error Mage::helper( 'turpentine/debug' )->logWarn( - 'Failed to parse param.show output to check esi_syntax value' ); + sprintf('Failed to parse param.show output to check %s value', $paramName ) ); $result = true; } } else { diff --git a/app/code/community/Nexcessnet/Turpentine/Model/Varnish/Admin/Socket.php b/app/code/community/Nexcessnet/Turpentine/Model/Varnish/Admin/Socket.php index db7580084..595b15384 100644 --- a/app/code/community/Nexcessnet/Turpentine/Model/Varnish/Admin/Socket.php +++ b/app/code/community/Nexcessnet/Turpentine/Model/Varnish/Admin/Socket.php @@ -84,10 +84,16 @@ class Nexcessnet_Turpentine_Model_Varnish_Admin_Socket { // and used const CLI_CMD_LENGTH_LIMIT = 8192; + /** + * Regexp to detect the varnish version number + * @var string + */ + const REGEXP_VARNISH_VERSION = '/^varnish\-(?P\d)\.(?P\d)\.(?P\d) revision (?P[0-9a-f]+)$/'; + /** * VCL config versions, should match config select values */ - static protected $_VERSIONS = array( '2.1', '3.0' ); + static protected $_VERSIONS = array( '2.1', '3.0', '4.0' ); /** * Varnish socket connection @@ -257,8 +263,8 @@ public function isConnected() { * @return string */ public function getVersion() { - if( is_null( $this->_version ) ) { - $this->_version = $this->_determineVersion(); + if ( !$this->isConnected() ) { + $this->_connect(); } return $this->_version; } @@ -331,15 +337,35 @@ protected function _connect() { $challenge = substr( $banner['text'], 0, 32 ); $response = hash( 'sha256', sprintf( "%s\n%s%s\n", $challenge, $this->_authSecret, $challenge ) ); - $this->_command( 'auth', self::CODE_OK, $response ); - } else if( $banner['code'] !== self::CODE_OK ) { + $banner = $this->_command( 'auth', self::CODE_OK, $response ); + } + + if( $banner['code'] !== self::CODE_OK ) { Mage::throwException( 'Varnish admin authentication failed: ' . $banner['text'] ); } + $this->_version = $this->_determineVersion($banner['text']); + return $this->isConnected(); } + protected function _determineVersion($bannerText) { + $bannerText = array_filter(explode("\n", $bannerText)); + if ( count($bannerText)<6 ) { + // Varnish 2.0 does not spit out a banner on connect + Mage::throwException('Varnish versions before 2.1 are not supported'); + } + if ( count($bannerText)<7 ) { + // Varnish before 3.0 does not spit out a version number + return '2.1'; + } elseif ( preg_match(self::REGEXP_VARNISH_VERSION, $bannerText[4], $matches)===1 ) { + return $matches['vmajor'] . '.' . $matches['vminor']; + } else { + Mage::throwException('Unable to detect varnish version'); + } + } + /** * Close the connection (if we're connected) * @@ -367,9 +393,17 @@ protected function _write( $data ) { $dataLength = strlen( $data ); if( $dataLength >= self::CLI_CMD_LENGTH_LIMIT ) { $cliBufferResponse = $this->param_show( 'cli_buffer' ); - if( preg_match( '~^cli_buffer\s+(\d+)\s+\[bytes\]~', - $cliBufferResponse['text'], $match ) ) { + $regexp = '~^cli_buffer\s+(\d+)\s+\[bytes\]~'; + if ( $this->getVersion()==='4.0' ) { + // Varnish4 supports "16k" style notation + $regexp = '~^cli_buffer\s+Value is:\s+(\d+)([k|m|g]{1})?\s+\[bytes\]~'; + } + if( preg_match( $regexp, $cliBufferResponse['text'], $match ) ) { $realLimit = (int)$match[1]; + if ( isset($match[2]) ) { + $factors = array('k'=>1,'m'=>2,'g'=>3); + $realLimit *= pow(1024, $factors[$match[2]]); + } } else { Mage::helper( 'turpentine/debug' )->logWarn( 'Failed to determine Varnish cli_buffer limit, using default' ); @@ -470,6 +504,7 @@ protected function _translateCommandMethod( $verb ) { case '2.1': $command = str_replace( 'ban', 'purge', $command ); break; + case '4.0': case '3.0': $command = str_replace( 'purge', 'ban', $command ); break; @@ -479,21 +514,4 @@ protected function _translateCommandMethod( $verb ) { } return $command; } - - /** - * Guess the Varnish version based on the availability of the 'banner' command - * - * @return string - */ - protected function _determineVersion() { - $resp = $this->_write( 'help' )->_read(); - if( strpos( $resp['text'], 'ban.url' ) !== false ) { - return '3.0'; - } elseif( strpos( $resp['text'], 'purge.url' ) !== false && - strpos( $resp['text'], 'banner' ) ) { - return '2.1'; - } else { - Mage::throwException( 'Unable to determine instance version' ); - } - } } diff --git a/app/code/community/Nexcessnet/Turpentine/Model/Varnish/Configurator/Abstract.php b/app/code/community/Nexcessnet/Turpentine/Model/Varnish/Configurator/Abstract.php index 5e29c3402..96dd7dce5 100644 --- a/app/code/community/Nexcessnet/Turpentine/Model/Varnish/Configurator/Abstract.php +++ b/app/code/community/Nexcessnet/Turpentine/Model/Varnish/Configurator/Abstract.php @@ -39,6 +39,11 @@ static public function getFromSocket( $socket ) { return null; } switch( $version ) { + case '4.0': + return Mage::getModel( + 'turpentine/varnish_configurator_version4', + array( 'socket' => $socket ) ); + case '3.0': return Mage::getModel( 'turpentine/varnish_configurator_version3', @@ -287,10 +292,14 @@ protected function _getDefaultBackend() { 'first_byte_timeout' => $timeout . 's', 'between_bytes_timeout' => $timeout . 's', ); - return $this->_vcl_backend( 'default', - Mage::getStoreConfig( 'turpentine_vcl/backend/backend_host' ), - Mage::getStoreConfig( 'turpentine_vcl/backend/backend_port' ), - $default_options ); + if ( Mage::getStoreConfigFlag( 'turpentine_vcl/backend/load_balancing' ) ) { + return $this->_vcl_director( 'default', $default_options ); + } else { + return $this->_vcl_backend( 'default', + Mage::getStoreConfig( 'turpentine_vcl/backend/backend_host' ), + Mage::getStoreConfig( 'turpentine_vcl/backend/backend_port' ), + $default_options ); + } } /** @@ -304,10 +313,14 @@ protected function _getAdminBackend() { 'first_byte_timeout' => $timeout . 's', 'between_bytes_timeout' => $timeout . 's', ); - return $this->_vcl_backend( 'admin', - Mage::getStoreConfig( 'turpentine_vcl/backend/backend_host' ), - Mage::getStoreConfig( 'turpentine_vcl/backend/backend_port' ), - $admin_options ); + if ( Mage::getStoreConfigFlag( 'turpentine_vcl/backend/load_balancing' ) ) { + return $this->_vcl_director( 'admin', $admin_options ); + } else { + return $this->_vcl_backend( 'admin', + Mage::getStoreConfig( 'turpentine_vcl/backend/backend_host' ), + Mage::getStoreConfig( 'turpentine_vcl/backend/backend_port' ), + $admin_options ); + } } /** @@ -587,6 +600,107 @@ protected function _vcl_backend( $name, $host, $port, $options=array() ) { return $str; } + /** + * Format a VCL director declaration, for load balancing + * + * @param string $name name of the director, also used to select config settings + * @param array $backendOptions options for each backend + * @return string + */ + protected function _vcl_director( $name, $backendOptions ) { + $tpl = <<cleanExplode( PHP_EOL, + Mage::getStoreConfig( 'turpentine_vcl/backend/backend_nodes_admin' ) ); + $probeUrl = Mage::getStoreConfig( 'turpentine_vcl/backend/backend_probe_url_admin' ); + } else { + $backendNodes = Mage::helper( 'turpentine/data' )->cleanExplode( PHP_EOL, + Mage::getStoreConfig( 'turpentine_vcl/backend/backend_nodes' ) ); + $probeUrl = Mage::getStoreConfig( 'turpentine_vcl/backend/backend_probe_url' ); + } + $backends = ''; + foreach ( $backendNodes as $backendNode ) { + $parts = explode( ':', $backendNode, 2 ); + $host = ( empty($parts[0]) ) ? '127.0.0.1' : $parts[0]; + $port = ( empty($parts[1]) ) ? '80' : $parts[1]; + $backends .= $this->_vcl_director_backend( $host, $port, $probeUrl, $backendOptions ); + } + $vars = array( + 'name' => $name, + 'backends' => $backends + ); + return $this->_formatTemplate( $tpl, $vars ); + } + + /** + * Format a VCL backend declaration to put inside director + * + * @param string $host backend host + * @param string $port backend port + * @param string $probeUrl URL to check if backend is up + * @param array $options extra options for backend + * @return string + */ + protected function _vcl_director_backend( $host, $port, $probeUrl='', $options=array() ) { + $tpl = << $host, + 'port' => $port, + 'probe' => '' + ); + if ( !empty( $probeUrl ) ) { + $vars['probe'] = $this->_vcl_get_probe( $probeUrl ); + } + $str = $this->_formatTemplate( $tpl, $vars ); + foreach( $options as $key => $value ) { + $str .= sprintf( ' .%s = %s;', $key, $value ) . PHP_EOL; + } + $str .= << $urlParts['host'], + 'probe_path' => $urlParts['path'] + ); + return $this->_formatTemplate( $tpl, $vars ); + } + } + /** * Format a VCL ACL declaration * diff --git a/app/code/community/Nexcessnet/Turpentine/Model/Varnish/Configurator/Version4.php b/app/code/community/Nexcessnet/Turpentine/Model/Varnish/Configurator/Version4.php new file mode 100644 index 000000000..6b5fe55d1 --- /dev/null +++ b/app/code/community/Nexcessnet/Turpentine/Model/Varnish/Configurator/Version4.php @@ -0,0 +1,60 @@ +_getVclTemplateFilename( self::VCL_TEMPLATE_FILE ); + $vcl = $this->_formatTemplate( file_get_contents( $tplFile ), + $this->_getTemplateVars() ); + return $doClean ? $this->_cleanVcl( $vcl ) : $vcl; + } + + // TODO: Check this + protected function _getAdvancedSessionValidation() { + $validation = ''; + foreach( $this->_getAdvancedSessionValidationTargets() as $target ) { + $validation .= sprintf( 'hash_data(%s);' . PHP_EOL, $target ); + } + return $validation; + } + + /** + * Build the list of template variables to apply to the VCL template + * + * @return array + */ + protected function _getTemplateVars() { + $vars = parent::_getTemplateVars(); + $vars['advanced_session_validation'] = + $this->_getAdvancedSessionValidation(); + return $vars; + } +} diff --git a/app/code/community/Nexcessnet/Turpentine/etc/config.xml b/app/code/community/Nexcessnet/Turpentine/etc/config.xml index a74f63beb..15a5c7420 100644 --- a/app/code/community/Nexcessnet/Turpentine/etc/config.xml +++ b/app/code/community/Nexcessnet/Turpentine/etc/config.xml @@ -49,8 +49,13 @@ + no 127.0.0.1 8080 + 127.0.0.1:8080 + + 127.0.0.1:8080 + 300 21600 127.0.0.1 @@ -131,6 +136,18 @@ Nexcessnet_Turpentine_Block_Poll_ActivePoll + + + + + Nexcessnet_Turpentine_Block_Product_Compared + Nexcessnet_Turpentine_Block_Product_Viewed + + + Nexcessnet_Turpentine_Block_Adminhtml_Cache_Grid diff --git a/app/code/community/Nexcessnet/Turpentine/etc/system.xml b/app/code/community/Nexcessnet/Turpentine/etc/system.xml index 822a05d7d..b6fc98715 100644 --- a/app/code/community/Nexcessnet/Turpentine/etc/system.xml +++ b/app/code/community/Nexcessnet/Turpentine/etc/system.xml @@ -239,6 +239,15 @@ 0 0 + + + select + turpentine/config_select_loadBalancing + 05 + 1 + 0 + 0 + text @@ -246,6 +255,9 @@ 1 0 0 + + no + @@ -255,7 +267,56 @@ 1 0 0 + + no + + + + textarea + A list of HOST:PORT pairs of the backend web servers, one per line + 22 + 1 + 0 + 0 + + yes|yes_admin + + + + + URL where Varnish can probe if a node is available. Leave empty to disable probing. + text + 23 + 1 + 0 + 0 + + yes|yes_admin + + + + + textarea + 25 + 1 + 0 + 0 + + yes_admin + + + + + text + 27 + 1 + 0 + 0 + + yes_admin + + text @@ -280,7 +341,7 @@ text 50 - Comma-separated list of IPs to serve cached pages to on initial visit + Comma-separated list of IP addresses that should bypass Turpentine's frontend cookie requirement. Also useful for performance testing tools as they also typically don't support cookies. Note - if you are using something like Pound to terminate SSL before Varnish, setting this to the same IP could cause users to end up sharing the crawler session. 1 0 0 diff --git a/app/code/community/Nexcessnet/Turpentine/misc/version-2.vcl b/app/code/community/Nexcessnet/Turpentine/misc/version-2.vcl index 3b9b8cc07..cb63ea02f 100644 --- a/app/code/community/Nexcessnet/Turpentine/misc/version-2.vcl +++ b/app/code/community/Nexcessnet/Turpentine/misc/version-2.vcl @@ -119,6 +119,9 @@ sub vcl_recv { if (req.http.X-Opt-Enable-Caching != "true" || req.http.Authorization || !(req.request ~ "^(GET|HEAD|OPTIONS)$") || req.http.Cookie ~ "varnish_bypass={{secret_handshake}}") { + if (req.url ~ "{{url_base_regex}}{{admin_frontname}}") { + set req.backend = admin; + } return (pipe); } diff --git a/app/code/community/Nexcessnet/Turpentine/misc/version-3.vcl b/app/code/community/Nexcessnet/Turpentine/misc/version-3.vcl index 6b33f4a07..0603b6bfe 100644 --- a/app/code/community/Nexcessnet/Turpentine/misc/version-3.vcl +++ b/app/code/community/Nexcessnet/Turpentine/misc/version-3.vcl @@ -121,6 +121,9 @@ sub vcl_recv { if (!{{enable_caching}} || req.http.Authorization || req.request !~ "^(GET|HEAD|OPTIONS)$" || req.http.Cookie ~ "varnish_bypass={{secret_handshake}}") { + if (req.url ~ "{{url_base_regex}}{{admin_frontname}}") { + set req.backend = admin; + } return (pipe); } diff --git a/app/code/community/Nexcessnet/Turpentine/misc/version-4.vcl b/app/code/community/Nexcessnet/Turpentine/misc/version-4.vcl new file mode 100644 index 000000000..89629ada7 --- /dev/null +++ b/app/code/community/Nexcessnet/Turpentine/misc/version-4.vcl @@ -0,0 +1,409 @@ +vcl 4.0; +# Nexcess.net Turpentine Extension for Magento +# Copyright (C) 2012 Nexcess.net L.L.C. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +## Nexcessnet_Turpentine Varnish v3 VCL Template + +## Custom C Code + +C{ + # @source app/code/community/Nexcessnet/Turpentine/misc/uuid.c + {{custom_c_code}} +}C + +## Imports + +import std; + +## Custom VCL Logic + +{{custom_vcl_include}} + +## Backends + +{{default_backend}} + +{{admin_backend}} + +## ACLs + +{{crawler_acl}} + +{{debug_acl}} + +## Custom Subroutines + +{{generate_session_start}} +sub generate_session { + # generate a UUID and add `frontend=$UUID` to the Cookie header, or use SID + # from SID URL param + if (req.url ~ ".*[&?]SID=([^&]+).*") { + set req.http.X-Varnish-Faked-Session = regsub( + req.url, ".*[&?]SID=([^&]+).*", "frontend=\1"); + } else { + C{ + char uuid_buf [50]; + generate_uuid(uuid_buf); + static const struct gethdr_s VGC_HDR_REQ_VARNISH_FAKED_SESSION = + { HDR_REQ, "\030X-Varnish-Faked-Session:"}; + VRT_SetHdr(ctx, + &VGC_HDR_REQ_VARNISH_FAKED_SESSION, + uuid_buf, + vrt_magic_string_end + ); + }C + } + if (req.http.Cookie) { + # client sent us cookies, just not a frontend cookie. try not to blow + # away the extra cookies + std.collect(req.http.Cookie); + set req.http.Cookie = req.http.X-Varnish-Faked-Session + + "; " + req.http.Cookie; + } else { + set req.http.Cookie = req.http.X-Varnish-Faked-Session; + } +} + +sub generate_session_expires { + # sets X-Varnish-Cookie-Expires to now + esi_private_ttl in format: + # Tue, 19-Feb-2013 00:14:27 GMT + # this isn't threadsafe but it shouldn't matter in this case + C{ + time_t now = time(NULL); + struct tm now_tm = *gmtime(&now); + now_tm.tm_sec += {{esi_private_ttl}}; + mktime(&now_tm); + char date_buf [50]; + strftime(date_buf, sizeof(date_buf)-1, "%a, %d-%b-%Y %H:%M:%S %Z", &now_tm); + static const struct gethdr_s VGC_HDR_RESP_COOKIE_EXPIRES = + { HDR_RESP, "\031X-Varnish-Cookie-Expires:"}; + VRT_SetHdr(ctx, + &VGC_HDR_RESP_COOKIE_EXPIRES, + date_buf, + vrt_magic_string_end + ); + }C +} +{{generate_session_end}} +## Varnish Subroutines + +sub vcl_recv { + # this always needs to be done so it's up at the top + if (req.restarts == 0) { + if (req.http.X-Forwarded-For) { + set req.http.X-Forwarded-For = + req.http.X-Forwarded-For + ", " + client.ip; + } else { + set req.http.X-Forwarded-For = client.ip; + } + } + + # We only deal with GET and HEAD by default + # we test this here instead of inside the url base regex section + # so we can disable caching for the entire site if needed + if (!{{enable_caching}} || req.http.Authorization || + req.method !~ "^(GET|HEAD|OPTIONS)$" || + req.http.Cookie ~ "varnish_bypass={{secret_handshake}}") { + return (pipe); + } + + # remove double slashes from the URL, for higher cache hit rate + set req.url = regsuball(req.url, "(.*)//+(.*)", "\1/\2"); + + {{normalize_encoding}} + {{normalize_user_agent}} + {{normalize_host}} + + # check if the request is for part of magento + if (req.url ~ "{{url_base_regex}}") { + # set this so Turpentine can see the request passed through Varnish + set req.http.X-Turpentine-Secret-Handshake = "{{secret_handshake}}"; + # use the special admin backend and pipe if it's for the admin section + if (req.url ~ "{{url_base_regex}}{{admin_frontname}}") { + set req.backend_hint = admin; + return (pipe); + } + if (req.http.Cookie ~ "\bcurrency=") { + set req.http.X-Varnish-Currency = regsub( + req.http.Cookie, ".*\bcurrency=([^;]*).*", "\1"); + } + if (req.http.Cookie ~ "\bstore=") { + set req.http.X-Varnish-Store = regsub( + req.http.Cookie, ".*\bstore=([^;]*).*", "\1"); + } + # looks like an ESI request, add some extra vars for further processing + if (req.url ~ "/turpentine/esi/get(?:Block|FormKey)/") { + set req.http.X-Varnish-Esi-Method = regsub( + req.url, ".*/{{esi_method_param}}/(\w+)/.*", "\1"); + set req.http.X-Varnish-Esi-Access = regsub( + req.url, ".*/{{esi_cache_type_param}}/(\w+)/.*", "\1"); + + # throw a forbidden error if debugging is off and a esi block is + # requested by the user (does not apply to ajax blocks) + if (req.http.X-Varnish-Esi-Method == "esi" && req.esi_level == 0 && + !({{debug_headers}} || client.ip ~ debug_acl)) { + return (synth(403, "External ESI requests are not allowed")); + } + } + # if host is not allowed in magento pass to backend + if (req.http.host !~ "{{allowed_hosts_regex}}") { + return (pass); + } + # no frontend cookie was sent to us AND this is not an ESI or AJAX call + if (req.http.Cookie !~ "frontend=" && !req.http.X-Varnish-Esi-Method) { + if (client.ip ~ crawler_acl || + req.http.User-Agent ~ "^(?:{{crawler_user_agent_regex}})$") { + # it's a crawler, give it a fake cookie + set req.http.Cookie = "frontend=crawler-session"; + } else { + # it's a real user, make up a new session for them + {{generate_session}}# call generate_session; + return (pipe); + } + } + if ({{force_cache_static}} && + req.url ~ ".*\.(?:{{static_extensions}})(?=\?|&|$)") { + # don't need cookies for static assets + unset req.http.Cookie; + unset req.http.X-Varnish-Faked-Session; + return (hash); + } + # this doesn't need a enable_url_excludes because we can be reasonably + # certain that cron.php at least will always be in it, so it will + # never be empty + if (req.url ~ "{{url_base_regex}}(?:{{url_excludes}})" || + # user switched stores. we pipe this instead of passing below because + # switching stores doesn't redirect (302), just acts like a link to + # another page (200) so the Set-Cookie header would be removed + req.url ~ "\?.*__from_store=") { + return (pipe); + } + if ({{enable_get_excludes}} && + req.url ~ "(?:[?&](?:{{get_param_excludes}})(?=[&=]|$))") { + # TODO: should this be pass or pipe? + return (pass); + } + if (req.url ~ "[?&](utm_source|utm_medium|utm_campaign|gclid|cx|ie|cof|siteurl)=") { + # Strip out Google related parameters + set req.url = regsuball(req.url, "(?:(\?)?|&)(?:utm_source|utm_medium|utm_campaign|gclid|cx|ie|cof|siteurl)=[^&]+", "\1"); + set req.url = regsuball(req.url, "(?:(\?)&|\?$)", "\1"); + } + + if ({{enable_get_ignored}} && req.url ~ "[?&]({{get_param_ignored}})=") { + # Strip out Ignored GET parameters + set req.url = regsuball(req.url, "(?:(\?)?|&)(?:{{get_param_ignored}})=[^&]+", "\1"); + set req.url = regsuball(req.url, "(?:(\?)&|\?$)", "\1"); + } + + # everything else checks out, try and pull from the cache + return (hash); + } + # else it's not part of magento so do default handling (doesn't help + # things underneath magento but we can't detect that) +} + +sub vcl_pipe { + # since we're not going to do any stuff to the response we pretend the + # request didn't pass through Varnish + unset bereq.http.X-Turpentine-Secret-Handshake; + set bereq.http.Connection = "close"; +} + +# sub vcl_pass { +# return (pass); +# } + +sub vcl_hash { + hash_data(req.url); + if (req.http.Host) { + hash_data(req.http.Host); + } else { + hash_data(server.ip); + } + hash_data(req.http.Ssl-Offloaded); + if (req.http.X-Normalized-User-Agent) { + hash_data(req.http.X-Normalized-User-Agent); + } + if (req.http.Accept-Encoding) { + # make sure we give back the right encoding + hash_data(req.http.Accept-Encoding); + } + if (req.http.X-Varnish-Store || req.http.X-Varnish-Currency) { + # make sure data is for the right store and currency based on the *store* + # and *currency* cookies + hash_data("s=" + req.http.X-Varnish-Store + "&c=" + req.http.X-Varnish-Currency); + } + + if (req.http.X-Varnish-Esi-Access == "private" && + req.http.Cookie ~ "frontend=") { + hash_data(regsub(req.http.Cookie, "^.*?frontend=([^;]*);*.*$", "\1")); + {{advanced_session_validation}} + + } + return (lookup); +} + +sub vcl_hit { + # this seems to cause cache object contention issues so removed for now + # TODO: use obj.hits % something maybe + # if (obj.hits > 0) { + # set obj.ttl = obj.ttl + {{lru_factor}}s; + # } +} + +# sub vcl_miss { +# return (fetch); +# } + +sub vcl_backend_response { + # set the grace period + set beresp.grace = {{grace_period}}s; + + # Store the URL in the response object, to be able to do lurker friendly bans later + set beresp.http.X-Varnish-Host = bereq.http.host; + set beresp.http.X-Varnish-URL = bereq.url; + + # if it's part of magento... + if (bereq.url ~ "{{url_base_regex}}") { + # we handle the Vary stuff ourselves for now, we'll want to actually + # use this eventually for compatibility with downstream proxies + # TODO: only remove the User-Agent field from this if it exists + unset beresp.http.Vary; + # we pretty much always want to do this + set beresp.do_gzip = true; + + if (beresp.status != 200 && beresp.status != 404) { + # pass anything that isn't a 200 or 404 + set beresp.ttl = {{grace_period}}s; + set beresp.uncacheable = true; + return (deliver); + } else { + # if Magento sent us a Set-Cookie header, we'll put it somewhere + # else for now + if (beresp.http.Set-Cookie) { + set beresp.http.X-Varnish-Set-Cookie = beresp.http.Set-Cookie; + unset beresp.http.Set-Cookie; + } + # we'll set our own cache headers if we need them + unset beresp.http.Cache-Control; + unset beresp.http.Expires; + unset beresp.http.Pragma; + unset beresp.http.Cache; + unset beresp.http.Age; + + if (beresp.http.X-Turpentine-Esi == "1") { + set beresp.do_esi = true; + } + if (beresp.http.X-Turpentine-Cache == "0") { + set beresp.ttl = {{grace_period}}s; + set beresp.uncacheable = true; + return (deliver); + } else { + if ({{force_cache_static}} && + bereq.url ~ ".*\.(?:{{static_extensions}})(?=\?|&|$)") { + # it's a static asset + set beresp.ttl = {{static_ttl}}s; + set beresp.http.Cache-Control = "max-age={{static_ttl}}"; + } elseif (bereq.http.X-Varnish-Esi-Method) { + # it's a ESI request + if (bereq.http.X-Varnish-Esi-Access == "private" && + bereq.http.Cookie ~ "frontend=") { + # set this header so we can ban by session from Turpentine + set beresp.http.X-Varnish-Session = regsub(bereq.http.Cookie, + "^.*?frontend=([^;]*);*.*$", "\1"); + } + if (bereq.http.X-Varnish-Esi-Method == "ajax" && + bereq.http.X-Varnish-Esi-Access == "public") { + set beresp.http.Cache-Control = "max-age=" + regsub( + bereq.url, ".*/{{esi_ttl_param}}/(\d+)/.*", "\1"); + } + set beresp.ttl = std.duration( + regsub( + bereq.url, ".*/{{esi_ttl_param}}/(\d+)/.*", "\1s"), + 300s); + if (beresp.ttl == 0s) { + # this is probably faster than bothering with 0 ttl + # cache objects + set beresp.ttl = {{grace_period}}s; + set beresp.uncacheable = true; + return (deliver); + } + } else { + {{url_ttls}} + } + } + } + # we've done what we need to, send to the client + return (deliver); + } + # else it's not part of Magento so use the default Varnish handling +} + +sub vcl_deliver { + if (req.http.X-Varnish-Faked-Session) { + # need to set the set-cookie header since we just made it out of thin air + {{generate_session_expires}} + set resp.http.Set-Cookie = req.http.X-Varnish-Faked-Session + + "; expires=" + resp.http.X-Varnish-Cookie-Expires + "; path=/"; + if (req.http.Host) { + if (req.http.User-Agent ~ "^(?:{{crawler_user_agent_regex}})$") { + # it's a crawler, no need to share cookies + set resp.http.Set-Cookie = resp.http.Set-Cookie + + "; domain=" + regsub(req.http.Host, ":\d+$", ""); + } else { + # it's a real user, allow sharing of cookies between stores + if(req.http.Host ~ "{{normalize_cookie_regex}}") { + set resp.http.Set-Cookie = resp.http.Set-Cookie + + "; domain={{normalize_cookie_target}}"; + } else { + set resp.http.Set-Cookie = resp.http.Set-Cookie + + "; domain=" + regsub(req.http.Host, ":\d+$", ""); + } + } + } + set resp.http.Set-Cookie = resp.http.Set-Cookie + "; httponly"; + unset resp.http.X-Varnish-Cookie-Expires; + } + if (req.http.X-Varnish-Esi-Method == "ajax" && req.http.X-Varnish-Esi-Access == "private") { + set resp.http.Cache-Control = "no-cache"; + } + if ({{debug_headers}} || client.ip ~ debug_acl) { + # debugging is on, give some extra info + set resp.http.X-Varnish-Hits = obj.hits; + set resp.http.X-Varnish-Esi-Method = req.http.X-Varnish-Esi-Method; + set resp.http.X-Varnish-Esi-Access = req.http.X-Varnish-Esi-Access; + set resp.http.X-Varnish-Currency = req.http.X-Varnish-Currency; + set resp.http.X-Varnish-Store = req.http.X-Varnish-Store; + } else { + # remove Varnish fingerprints + unset resp.http.X-Varnish; + unset resp.http.Via; + unset resp.http.X-Powered-By; + unset resp.http.Server; + unset resp.http.X-Turpentine-Cache; + unset resp.http.X-Turpentine-Esi; + unset resp.http.X-Turpentine-Flush-Events; + unset resp.http.X-Turpentine-Block; + unset resp.http.X-Varnish-Session; + unset resp.http.X-Varnish-Host; + unset resp.http.X-Varnish-URL; + # this header indicates the session that originally generated a cached + # page. it *must* not be sent to a client in production with lax + # session validation or that session can be hijacked + unset resp.http.X-Varnish-Set-Cookie; + } +} diff --git a/app/design/frontend/base/default/layout/turpentine_esi.xml b/app/design/frontend/base/default/layout/turpentine_esi.xml index 1ca2f9b6e..f44357b68 100644 --- a/app/design/frontend/base/default/layout/turpentine_esi.xml +++ b/app/design/frontend/base/default/layout/turpentine_esi.xml @@ -88,6 +88,8 @@ private + + @@ -240,6 +242,22 @@ + + + + + + + + + + + + + + + + diff --git a/app/design/frontend/base/default/template/turpentine/ajax.phtml b/app/design/frontend/base/default/template/turpentine/ajax.phtml index 7248695dc..4756c9f55 100644 --- a/app/design/frontend/base/default/template/turpentine/ajax.phtml +++ b/app/design/frontend/base/default/template/turpentine/ajax.phtml @@ -32,6 +32,9 @@ if( $debugEnabled ) { * be an issue we'll have to go back to using Ajax.Request so the container * block is completely replaced which means no nice appear effect. * + * The 10 ms delay after page load is to make sure the Ajax call is + * executed async so it does not delay DOMContentLoaded. Better for Google. + * * @link http://prototypejs.org/doc/latest/ajax/index.html * @link http://prototypejs.org/doc/latest/ajax/Ajax/Request/index.html * @link http://prototypejs.org/doc/latest/dom/Element/replace/index.html @@ -47,30 +50,34 @@ echo << (function() { var blockTag = {$this->helper('core')->jsonEncode($blockTag)}, esiUrl = {$this->helper('core')->jsonEncode($this->getEsiUrl())}; - if (typeof Ajax === 'object' && typeof Ajax.Updater === 'function') { - new Ajax.Updater( - blockTag, - esiUrl, - { - method: "get", - evalScripts: true, - {$_prototypeFunction}: function() { - $(blockTag).appear({ - duration: 0.3 - }); + if (typeof Ajax === 'object' && typeof Ajax.Updater === 'function' && typeof Event === 'function' ) { + Event.observe( window, "load", function() { setTimeout( function() { + new Ajax.Updater( + blockTag, + esiUrl, + { + method: "get", + evalScripts: true, + {$_prototypeFunction}: function() { + $(blockTag).appear({ + duration: 0.3 + }); + } } - } - ); + ); + }, 10 ); } ); } else if (typeof jQuery === 'function') { - jQuery.ajax( - { - url: esiUrl, - type: "get", - dataType: "html" - } - ).{$_jQueryFunction}(function() { - $(blockTag).fadeIn(300); - }); + jQuery(document).ready( function() { setTimeout( function() { + jQuery.ajax( + { + url: esiUrl, + type: "get", + dataType: "html" + } + ).{$_jQueryFunction}(function(data) { + jQuery('#'+blockTag).html(data).fadeIn(300); + }); + }, 10 ); } ); } })();