[{"data":1,"prerenderedAt":199},["ShallowReactive",2],{"post-/posts/bypassing-server-cache":3},{"id":4,"title":5,"body":6,"created":189,"description":190,"extension":191,"meta":192,"navigation":193,"path":194,"seo":195,"stem":196,"updated":197,"__hash__":198},"posts/2.posts/20230807.Bypassing Server Cache.md","Bypassing server cache when digests don't match",{"type":7,"value":8,"toc":181},"minimark",[9,13,36,39,50,55,58,61,74,78,87,91,103,106,110,119,129,132,138,145,156,168,172,175,178],[10,11,12],"p",{},"I'm building an application that keeps track of a file that is hosted somewhere on a public web server (which is out of my control).",[10,14,15,16,20,21,28,29,32,33,35],{},"The web server serves the file that I'm interested in, which we'll call ",[17,18,19],"code",{},"cool-file.txt",", along with an ",[22,23,27],"a",{"href":24,"rel":25},"https://en.wikipedia.org/wiki/MD5",[26],"nofollow","MD5 digest"," of the file called ",[17,30,31],{},"cool-file.txt.md5",".\nThis MD5 digest is like a small 'fingerprint' of the actual file.\n",[17,34,19],{}," could be many megabytes in size, but the fingerprint is only a few bytes.",[10,37,38],{},"MD5 digests like this are usually used for verifying the integrity of downloaded files: After downloading the file, you can compute the digest yourself, and check if it matches the digest from the server.",[10,40,41,42,44,45,49],{},"Another reason for using the digest is to save on resources.\nIf you want to know if ",[17,43,19],{}," has changed, you can simply retrieve the (small) digest, and see if ",[46,47,48],"em",{},"that"," changed. This way you don't have to redownload the (potentially huge) file every time you want to check if it changed.",[51,52,54],"h2",{"id":53},"the-problem","The problem",[10,56,57],{},"However, I noticed that when retrieving the file and computing the MD5 digest myself, it did not match the digest from the server!",[10,59,60],{},"This is problematic because of two reasons:",[62,63,64,68],"ol",{},[65,66,67],"li",{},"My application's integrity validation fails because my digest does not match the server's digest.",[65,69,70,71,73],{},"When ",[17,72,19],{}," changes, my application wouldn't notice it because it will keep receiving the old digest.",[51,75,77],{"id":76},"the-cause","The cause",[10,79,80,81,86],{},"It turned out that the file was recently updated, but I did not yet receive the new MD5 digest because the old digest was cached at the web server's end.\nThat is to say: the web server receives a request for the digest, and as an optimization, it doesn't read the updated digest from its file system (",[22,82,85],{"href":83,"rel":84},"https://atp.fm/",[26],"🛎️","), but it returns the old digest that it still memorized!",[51,88,90],{"id":89},"fix-attempt-1","Fix attempt #1 ❌",[10,92,93,94,102],{},"Unfortunately, trying to bypass the cache with a ",[22,95,98,101],{"href":96,"rel":97},"https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control",[26],[17,99,100],{},"\"Cache-Control: no-cache\""," header"," did not work, because the server did not respect that header.",[10,104,105],{},"Ideally, when a server receives a request with that header included, it should disable the cache and give the latest result. But a server can choose to ignore it.",[51,107,109],{"id":108},"fix-attempt-2","Fix attempt #2 ✅",[10,111,112,113,118],{},"I tried adding a ",[22,114,117],{"href":115,"rel":116},"https://www.rfc-editor.org/rfc/rfc3986.html#section-3.4",[26],"query component"," to the URL.\nThis component is ignored by most (if not all) web servers when serving static files.\nSo, instead of",[120,121,126],"pre",{"className":122,"code":124,"language":125},[123],"language-text","https://example.org/cool-file.txt.md5\n","text",[17,127,124],{"__ignoreMap":128},"",[10,130,131],{},"I used",[120,133,136],{"className":134,"code":135,"language":125},[123],"https://example.org/cool-file.txt.md5?hello\n",[17,137,135],{"__ignoreMap":128},[10,139,140,141,144],{},"and it worked!\nApparently, the cache in the web server works with some sort of lookup table that allows it to look up the memorized responses for certain URLs (",[46,142,143],{},"including"," the query component).",[10,146,147,148,151,152,155],{},"Of course, for this fix to work consistently, the part after the ",[17,149,150],{},"?"," should be different every time, because otherwise the web server can just return another outdated digest that it memorized for the ",[17,153,154],{},"https://example.org/cool-file.txt.md5?hello"," request.",[10,157,158,159,161,162,167],{},"For this reason, my application now just appends ",[17,160,150],{}," and a ",[22,163,166],{"href":164,"rel":165},"https://developer.mozilla.org/en-US/docs/Glossary/UUID",[26],"UUID"," to the URL of the digest (and the file itself as well!).",[51,169,171],{"id":170},"discussion","Discussion",[10,173,174],{},"This doesn't feel like a great fix.\nWith every request I do now, this will probably create an entry in the cache in the web server which consumes memory on their end.",[10,176,177],{},"But hey, the cache at the web server is out of my control, so I had to make do with it somehow!",[10,179,180],{},"What do you think? Should I contact the administrator of the web server to better invalidate their caches?",{"title":128,"searchDepth":182,"depth":182,"links":183},2,[184,185,186,187,188],{"id":53,"depth":182,"text":54},{"id":76,"depth":182,"text":77},{"id":89,"depth":182,"text":90},{"id":108,"depth":182,"text":109},{"id":170,"depth":182,"text":171},"2023-08-07","How to bypass server cache with a URL query component.","md",{},true,"/posts/bypassing-server-cache",{"title":5,"description":190},"2.posts/20230807.Bypassing Server Cache",null,"d6NiEFAtm26tdS_C7-uDi9DvDCSn_U-lHUim8Qm8_MU",1775234851922]