{"id":149,"date":"2009-05-27T22:37:50","date_gmt":"2009-05-28T03:37:50","guid":{"rendered":"http:\/\/www.1oc.com\/blog\/?p=149"},"modified":"2009-05-27T22:37:50","modified_gmt":"2009-05-28T03:37:50","slug":"creating-a-static-copy-of-a-dynamic-website","status":"publish","type":"post","link":"http:\/\/blog.1oc.com\/?p=149","title":{"rendered":"Creating a static copy of a dynamic website"},"content":{"rendered":"

\u00a0<\/p>\n

At work we have several websites that we develop with Plone, but each year we make a new version and we want to keep an archive of the old version.<\/p>\n

Since it takes a lot of memory to keep a Zope instance for these old websites that probably won\u2019t need to be edited ever again, it makes sense to make a static copy of the website. It also eliminates the work needed to update the instance when security patches come out (and eliminates security risks, in cases of old versions that are no more maintained).<\/p>\n

There are some tools that can help in this case; I chose to use wget, which is available in most Linux distributions by default.<\/p>\n

\u00a0<\/p>\n

The command line, in short\u2026<\/strong><\/p>\n

\u00a0<\/p>\n

wget -k -K  -E -r -l 10 -p -N -F --restrict-file-names=windows -nH http:\/\/website.com\/<\/a><\/pre>\n
\u2026and the options explained\n<\/strong><\/pre>\n
-k : convert links to relative\n-K : keep an original versions of files without the conversions made by wget\n-E : rename html files to .html (if they don\u2019t already have an htm(l) extension)\n-r : recursive\u2026 of course we want to make a recursive copy\n-l 10 : the maximum level of recursion. if you have a really big website you may need to put a higher number, but 10 levels should be enough.\n-p : download all necessary files for each page (css, js, images)\n-N : Turn on time-stamping.\n-F : When input is read from a file, force it to be treated as an HTML file.\n-nH : By default, wget put files in a directory named after the site\u2019s hostname. This will disabled creating of those hostname directories and put everything in the current directory.\n\u2013restrict-file-names=windows : may be useful if you want to copy the files to a Windows PC.<\/pre>\n
<\/pre>\n
Possible problems\n<\/strong><\/pre>\n
<\/pre>\n
\n\t
  • wget download the homagepage, robots.txt then stops!\nYour robots.txt file probably denies access to your site to search engines. Yes, in recursive mode, wget will respect the robots.txt file, so you will need to remove it before making the copy. Don\u2019t forget to put it back in the static site if that\u2019s what you want.<\/li>\n\t
  • Stylesheets : if you have @import stylesheet imports, wget won\u2019t see them, and won\u2019t download them\u00a0\":(\"\u00a0You might want to change them to <link rel=\u201dstylesheet\u201d \u2026 \/> imports, which wget will see and download.<\/li>\n\t
  • Stylesheet images : wget won\u2019t download background-images referenced in CSS files. For most websites that should not be too long to download those images manually.<\/li>\n\t
  • Be sure that you CSS files and with \u201c.css\u201d! Apache won\u2019t send the correct mime-type if your file extension is not .css, and Firefox will not use the stylesheet.\n(test.css?color=blue won\u2019t work, change it to test.css?color=blue&ext=.css)\nThe same problem may happen with other files types that need to have a proper mimetype set (video files, for instance)<\/li>\n\t
  • LinguaPlone specific problems<\/strong>\n