Web Crawler

{ "type": "excalidraw", "version": 2, "source": "https://marketplace.visualstudio.com/items?itemName=pomdtr.excalidraw-editor", "elements": [ { "id": "FWHq7bG44yS9BuJ3hICuo", "type": "text", "x": 319.29923727141954, "y": 159.10327002555755, "width": 547.57958984375, "height": 150, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a1", "roundness": null, "seed": 192892474, "version": 275, "versionNonce": 963425191, "isDeleted": false, "boundElements": [], "updated": 1766974296152, "link": null, "locked": false, "text": "Functional requirements\n\n- Crawl the web starting from a given set of seed URLs\n- Extract text data from each web page and store \nthe text for later processing\n- Add new URLs to the list of URLs to be downloaded", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Functional requirements\n\n- Crawl the web starting from a given set of seed URLs\n- Extract text data from each web page and store \nthe text for later processing\n- Add new URLs to the list of URLs to be downloaded", "autoResize": true, "lineHeight": 1.25 }, { "id": "j1kPaSB3CPp9kwDEmhiHV", "type": "text", "x": 1007.9607767733823, "y": 132.13962177668085, "width": 695.5595092773438, "height": 200, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a3", "roundness": null, "seed": 1128128442, "version": 351, "versionNonce": 1182899527, "isDeleted": false, "boundElements": [], "updated": 1766974309067, "link": null, "locked": false, "text": "Non Functional Requirements\n\n- Fault tolerance to handle failures gracefully and resume crawling \nwithout losing progress\n- Politeness to adhere to robots.txt and not overload website servers\ninappropriately\n- Efficiency to crawl the web in under 5 days\n- Scale ot 10B pages", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Non Functional Requirements\n\n- Fault tolerance to handle failures gracefully and resume crawling \nwithout losing progress\n- Politeness to adhere to robots.txt and not overload website servers\ninappropriately\n- Efficiency to crawl the web in under 5 days\n- Scale ot 10B pages", "autoResize": true, "lineHeight": 1.25 }, { "id": "7pTmW3sqvXS8PVgOBwUHp", "type": "text", "x": 317.2842779236303, "y": 375.7910039373096, "width": 450.6197509765625, "height": 300, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a5", "roundness": null, "seed": 1414321658, "version": 294, "versionNonce": 498932393, "isDeleted": false, "boundElements": [], "updated": 1766974313703, "link": null, "locked": false, "text": "Back of envelope estimation\n\nAssume 1b web pages \nQPS: 1000000000/30 days/24 hrs/3600 sec\n 400 pages per second\nPeak QPS: 2*QPS=800\n\nWeb page size is 500k\n1b x 500k=500TB per month\n\nfor 5 years\n500TB*12 months*5 years=30PB", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Back of envelope estimation\n\nAssume 1b web pages \nQPS: 1000000000/30 days/24 hrs/3600 sec\n 400 pages per second\nPeak QPS: 2*QPS=800\n\nWeb page size is 500k\n1b x 500k=500TB per month\n\nfor 5 years\n500TB*12 months*5 years=30PB", "autoResize": true, "lineHeight": 1.25 }, { "id": "bOGU4awJMy7nkfRNrm-3Y", "type": "rectangle", "x": 1361.8139456493182, "y": 580.4117324297769, "width": 252.53151416087417, "height": 101.66391595518462, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a6", "roundness": { "type": 3 }, "seed": 1465771322, "version": 43, "versionNonce": 795306278, "isDeleted": false, "boundElements": [ { "id": "CE5LJHfAYBUjRx-1Xm1j9", "type": "arrow" } ], "updated": 1760910773773, "link": null, "locked": false }, { "id": "_7PJS4Ckxx2nIG38rgeZP", "type": "rectangle", "x": 2250.972340344677, "y": 581.8994970535109, "width": 252.53151416087417, "height": 101.66391595518462, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a8", "roundness": { "type": 3 }, "seed": 1477480954, "version": 129, "versionNonce": 267769446, "isDeleted": false, "boundElements": [ { "id": "lLGxi0fddNrH2ilMeYto9", "type": "arrow" }, { "id": "4-5EjKJ7iDaFjefgdMcAB", "type": "arrow" }, { "id": "oRTB8YKUF69jA9nG89oxL", "type": "arrow" } ], "updated": 1760910867197, "link": null, "locked": false }, { "id": "r4zdO3AeoIeIMouca-63V", "type": "rectangle", "x": 1796.9727000531016, "y": 581.9738852846978, "width": 252.53151416087417, "height": 101.66391595518462, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a9", "roundness": { "type": 3 }, "seed": 14496038, "version": 132, "versionNonce": 283172922, "isDeleted": false, "boundElements": [ { "id": "CE5LJHfAYBUjRx-1Xm1j9", "type": "arrow" }, { "id": "lLGxi0fddNrH2ilMeYto9", "type": "arrow" }, { "id": "mE4q3BA-BlIRxHfDKhFhF", "type": "arrow" } ], "updated": 1760910841785, "link": null, "locked": false }, { "id": "5PG3U6xwgIZIJUh0rIBvL", "type": "rectangle", "x": 2249.4845757209428, "y": 353.61854625950775, "width": 252.53151416087417, "height": 101.66391595518462, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aA", "roundness": { "type": 3 }, "seed": 227588390, "version": 87, "versionNonce": 1326707942, "isDeleted": false, "boundElements": [ { "id": "oRTB8YKUF69jA9nG89oxL", "type": "arrow" } ], "updated": 1760910867197, "link": null, "locked": false }, { "id": "vvxEtSQGVRNOhrW1nS4oM", "type": "rectangle", "x": 2732.131950378426, "y": 577.3948763872042, "width": 252.53151416087417, "height": 101.66391595518462, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aB", "roundness": { "type": 3 }, "seed": 537330554, "version": 90, "versionNonce": 1564464634, "isDeleted": false, "boundElements": [ { "id": "4-5EjKJ7iDaFjefgdMcAB", "type": "arrow" }, { "id": "UoQP_GcjhECmQFTI7CtQB", "type": "arrow" } ], "updated": 1760910793923, "link": null, "locked": false }, { "id": "eKWO_j7vSgr7VHYeJHm4p", "type": "rectangle", "x": 3269.6199821385635, "y": 578.3453926745902, "width": 252.53151416087417, "height": 101.66391595518462, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aC", "roundness": { "type": 3 }, "seed": 561469094, "version": 97, "versionNonce": 959731366, "isDeleted": false, "boundElements": [ { "id": "UoQP_GcjhECmQFTI7CtQB", "type": "arrow" }, { "id": "STwreyezVBMh271RYAskG", "type": "arrow" }, { "id": "4AGn9EuWehrMlqSE-HUHF", "type": "arrow" }, { "id": "e_w5y2WbidAQpuK-Ynli2", "type": "arrow" }, { "id": "DojGmf3EC0T5LGC_8oCqw", "type": "arrow" } ], "updated": 1760910974766, "link": null, "locked": false }, { "id": "SlxgiELYUsc-R4f1IQam_", "type": "rectangle", "x": 3312.343622916805, "y": 1127.2148238062964, "width": 252.53151416087417, "height": 101.66391595518462, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aD", "roundness": { "type": 3 }, "seed": 1788365882, "version": 145, "versionNonce": 985280870, "isDeleted": false, "boundElements": [ { "id": "uHS8SuSeMm9Lj0_hjxmWj", "type": "arrow" }, { "id": "cbsmQpob6F5TE46EH3v9O", "type": "arrow" } ], "updated": 1760910825007, "link": null, "locked": false }, { "id": "VyNrta42IqLTjEbY2qqBH", "type": "rectangle", "x": 3299.1190484836093, "y": 1354.6609733392047, "width": 252.53151416087417, "height": 101.66391595518462, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aE", "roundness": { "type": 3 }, "seed": 953010918, "version": 146, "versionNonce": 1408791738, "isDeleted": false, "boundElements": [ { "id": "uHS8SuSeMm9Lj0_hjxmWj", "type": "arrow" }, { "id": "mE4q3BA-BlIRxHfDKhFhF", "type": "arrow" }, { "id": "QV7JTfuXAHsrO_e5f0qPv", "type": "arrow" } ], "updated": 1760910979348, "link": null, "locked": false }, { "id": "R0doMiiW1yTOfysXkOoFl", "type": "rectangle", "x": 2980.861399389757, "y": 824.2232928237635, "width": 252.53151416087417, "height": 101.66391595518462, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aF", "roundness": { "type": 3 }, "seed": 2076585402, "version": 46, "versionNonce": 1122975546, "isDeleted": false, "boundElements": [ { "id": "4AGn9EuWehrMlqSE-HUHF", "type": "arrow" } ], "updated": 1760910848997, "link": null, "locked": false }, { "id": "djLV8BYZoomMC32lSeiEc", "type": "rectangle", "x": 3287.1094818264637, "y": 823.5951255381868, "width": 252.53151416087417, "height": 101.66391595518462, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aG", "roundness": { "type": 3 }, "seed": 1419583846, "version": 47, "versionNonce": 1925025766, "isDeleted": false, "boundElements": [ { "id": "STwreyezVBMh271RYAskG", "type": "arrow" }, { "id": "cbsmQpob6F5TE46EH3v9O", "type": "arrow" } ], "updated": 1760910825007, "link": null, "locked": false }, { "id": "6SBTNyQwbXMsAxVLzMQvd", "type": "rectangle", "x": 3670.3245874643644, "y": 817.4704945038133, "width": 252.53151416087417, "height": 101.66391595518462, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aH", "roundness": { "type": 3 }, "seed": 634411386, "version": 82, "versionNonce": 482350950, "isDeleted": false, "boundElements": [ { "id": "e_w5y2WbidAQpuK-Ynli2", "type": "arrow" } ], "updated": 1760910855515, "link": null, "locked": false }, { "id": "CE5LJHfAYBUjRx-1Xm1j9", "type": "arrow", "x": 1618.2880360630888, "y": 636.5169894626056, "width": 182.08585922705015, "height": 0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aI", "roundness": { "type": 2 }, "seed": 588857018, "version": 34, "versionNonce": 506156134, "isDeleted": false, "boundElements": [], "updated": 1760910773773, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 182.08585922705015, 0 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "bOGU4awJMy7nkfRNrm-3Y", "focus": 0.10373983739837264, "gap": 3.942576252896515 }, "endBinding": { "elementId": "r4zdO3AeoIeIMouca-63V", "focus": -0.07300813008130011, "gap": 3.4011952370374274 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "lLGxi0fddNrH2ilMeYto9", "type": "arrow", "x": 2050.7092378365196, "y": 631.9474636142921, "width": 193.60481742691718, "height": 0.5972641010641837, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aK", "roundness": { "type": 2 }, "seed": 1354991994, "version": 57, "versionNonce": 633554278, "isDeleted": false, "boundElements": [], "updated": 1760910862677, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 193.60481742691718, 0.5972641010641837 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "r4zdO3AeoIeIMouca-63V", "focus": -0.02443549993843816, "gap": 1.3017689505195449 }, "endBinding": { "elementId": "_7PJS4Ckxx2nIG38rgeZP", "focus": -0.00436025467540453, "gap": 7.261142994974307 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "4-5EjKJ7iDaFjefgdMcAB", "type": "arrow", "x": 2510.932983575111, "y": 633.7369749870592, "width": 214.68310051848857, "height": 6.434344686468194, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aL", "roundness": { "type": 2 }, "seed": 1609257254, "version": 56, "versionNonce": 1296148646, "isDeleted": false, "boundElements": [], "updated": 1760910862677, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 214.68310051848857, -6.434344686468194 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "_7PJS4Ckxx2nIG38rgeZP", "focus": -0.10679944166041008, "gap": 8.112424783614188 }, "endBinding": { "elementId": "vvxEtSQGVRNOhrW1nS4oM", "focus": 0.018181555481552456, "gap": 7.104101173580602 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "UoQP_GcjhECmQFTI7CtQB", "type": "arrow", "x": 2984.427876711961, "y": 628.5589648717443, "width": 295.3708699654044, "height": 1.0331698775934228, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aM", "roundness": { "type": 2 }, "seed": 1702751610, "version": 94, "versionNonce": 197729638, "isDeleted": false, "boundElements": [], "updated": 1760910797588, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 295.3708699654044, 1.0331698775934228 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "vvxEtSQGVRNOhrW1nS4oM", "focus": -0.00212013618414742, "gap": 1 }, "endBinding": { "elementId": "eKWO_j7vSgr7VHYeJHm4p", "focus": -0.016009045089566225, "gap": 10.178764538801715 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "STwreyezVBMh271RYAskG", "type": "arrow", "x": 3399.9192344211265, "y": 680.0754315019408, "width": 0.6529633626391842, "height": 149.55340612139105, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aO", "roundness": { "type": 2 }, "seed": 2066480550, "version": 48, "versionNonce": 1435812474, "isDeleted": false, "boundElements": [], "updated": 1760910809880, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 0.6529633626391842, 149.55340612139105 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "eKWO_j7vSgr7VHYeJHm4p", "focus": -0.03013154540434814, "gap": 1 }, "endBinding": { "elementId": "djLV8BYZoomMC32lSeiEc", "focus": -0.09967331715915348, "gap": 6.033712085144998 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "cbsmQpob6F5TE46EH3v9O", "type": "arrow", "x": 3412.3783416730917, "y": 921.4398588550175, "width": 0.6529633626391842, "height": 195.2195147110167, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aP", "roundness": { "type": 2 }, "seed": 1705139386, "version": 132, "versionNonce": 1234084646, "isDeleted": false, "boundElements": [], "updated": 1760910825007, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 0.6529633626391842, 195.2195147110167 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "djLV8BYZoomMC32lSeiEc", "focus": 0.009128301564019373, "gap": 3.8191826383539365 }, "endBinding": { "elementId": "SlxgiELYUsc-R4f1IQam_", "focus": -0.2006769675108376, "gap": 10.55545024026219 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "uHS8SuSeMm9Lj0_hjxmWj", "type": "arrow", "x": 3432.802043813356, "y": 1221.0425926390442, "width": 0.6446980036175773, "height": 135.37831540081038, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aQ", "roundness": { "type": 2 }, "seed": 213971558, "version": 73, "versionNonce": 1714863846, "isDeleted": false, "boundElements": [], "updated": 1760910822908, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -0.6446980036175773, 135.37831540081038 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "SlxgiELYUsc-R4f1IQam_", "focus": 0.044286442825166035, "gap": 7.8361471224368415 }, "endBinding": { "elementId": "VyNrta42IqLTjEbY2qqBH", "focus": 0.05168731247297531, "gap": 1.7599347006498647 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "mE4q3BA-BlIRxHfDKhFhF", "type": "arrow", "x": 3294.1190484836093, "y": 1405.392931316797, "width": 1387.36998259646, "height": 714.8700860126326, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aR", "roundness": null, "seed": 1301525242, "version": 136, "versionNonce": 29088678, "isDeleted": false, "boundElements": [], "updated": 1760910841791, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -1387.36998259646, 0 ], [ -1387.36998259646, -714.8700860126326 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "VyNrta42IqLTjEbY2qqBH", "fixedPoint": [ -0.019799509049847817, 0.4990163668292692 ], "focus": 0, "gap": 0 }, "endBinding": { "elementId": "r4zdO3AeoIeIMouca-63V", "fixedPoint": [ 0.43470362975812654, 1.0677235772357738 ], "focus": 0, "gap": 0 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": true, "fixedSegments": null, "startIsSpecial": null, "endIsSpecial": null }, { "id": "4AGn9EuWehrMlqSE-HUHF", "type": "arrow", "x": 3395.785739219001, "y": 685.0093086297749, "width": 288.7585827488065, "height": 134.2139841939886, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aS", "roundness": null, "seed": 1112372966, "version": 31, "versionNonce": 1543294118, "isDeleted": false, "boundElements": [], "updated": 1760910850347, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -288.7585827488065, 134.2139841939886 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "eKWO_j7vSgr7VHYeJHm4p", "fixedPoint": [ 0.4996040098190039, 1.0491816585365854 ], "focus": 0, "gap": 0 }, "endBinding": { "elementId": "R0doMiiW1yTOfysXkOoFl", "fixedPoint": [ 0.4996040098190039, -0.049181658536585335 ], "focus": 0, "gap": 0 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false, "fixedSegments": null, "startIsSpecial": null, "endIsSpecial": null }, { "id": "e_w5y2WbidAQpuK-Ynli2", "type": "arrow", "x": 3399.62168149638, "y": 682.05911766692, "width": 382.3720390177841, "height": 131.6671692004951, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aT", "roundness": null, "seed": 1766344678, "version": 39, "versionNonce": 974687270, "isDeleted": false, "boundElements": [], "updated": 1760910855515, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 382.3720390177841, 131.6671692004951 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "eKWO_j7vSgr7VHYeJHm4p", "focus": 0.5470781336348508, "gap": 2.04980903714511 }, "endBinding": { "elementId": "6SBTNyQwbXMsAxVLzMQvd", "focus": 0.5253902219435653, "gap": 3.744207636398187 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "oRTB8YKUF69jA9nG89oxL", "type": "arrow", "x": 2371.2457121350744, "y": 582.9409322901254, "width": 3.653288687170061, "height": 124.45151077538321, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aV", "roundness": null, "seed": 12023162, "version": 34, "versionNonce": 1425676710, "isDeleted": false, "boundElements": [], "updated": 1760910867197, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -3.653288687170061, -124.45151077538321 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "_7PJS4Ckxx2nIG38rgeZP", "focus": -0.0354637749319002, "gap": 1.0414352366144612 }, "endBinding": { "elementId": "5PG3U6xwgIZIJUh0rIBvL", "focus": 0.07627102059883832, "gap": 3.206959300049789 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "02foHHEyRGOeqb2h-nBrw", "type": "text", "x": 1427.23426229853, "y": 616.4486977602326, "width": 103.47990417480469, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aW", "roundness": null, "seed": 2045121850, "version": 31, "versionNonce": 1100501030, "isDeleted": false, "boundElements": [], "updated": 1760910877622, "link": null, "locked": false, "text": "seed URLs", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "seed URLs", "autoResize": true, "lineHeight": 1.25 }, { "id": "ZIW3-0AIYX-7pESm31skc", "type": "text", "x": 1854.090463565973, "y": 625.2843665534109, "width": 127.07989501953125, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aX", "roundness": null, "seed": 228259322, "version": 23, "versionNonce": 1360955750, "isDeleted": false, "boundElements": [], "updated": 1760910887540, "link": null, "locked": false, "text": "URL Frontier", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "URL Frontier", "autoResize": true, "lineHeight": 1.25 }, { "id": "ONEhe8BmTc7HPedUwq1qN", "type": "text", "x": 2283.7651522594915, "y": 628.9955127537261, "width": 173.0598907470703, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aY", "roundness": null, "seed": 515020986, "version": 25, "versionNonce": 1991314106, "isDeleted": false, "boundElements": [], "updated": 1760910897600, "link": null, "locked": false, "text": "HTML Downloader", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "HTML Downloader", "autoResize": true, "lineHeight": 1.25 }, { "id": "FvuknqnxaY4hBHsr3QFWJ", "type": "text", "x": 2279.3845119784955, "y": 406.21102570851923, "width": 130.97991943359375, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aZ", "roundness": null, "seed": 1957030182, "version": 20, "versionNonce": 312141542, "isDeleted": false, "boundElements": [], "updated": 1760910906350, "link": null, "locked": false, "text": "DNS Resolver", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "DNS Resolver", "autoResize": true, "lineHeight": 1.25 }, { "id": "9ghWj5UwIAVDsBXMFEHOS", "type": "text", "x": 2786.117142822435, "y": 632.8967622115185, "width": 149.23988342285156, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aa", "roundness": null, "seed": 1286177594, "version": 16, "versionNonce": 1036430970, "isDeleted": false, "boundElements": [], "updated": 1760910914200, "link": null, "locked": false, "text": "Content Parser", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Content Parser", "autoResize": true, "lineHeight": 1.25 }, { "id": "N4oXtLUf-yrFAuNmduaTV", "type": "text", "x": 3358.3775399828546, "y": 622.7634320520831, "width": 140.23989868164062, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ab", "roundness": null, "seed": 484727142, "version": 15, "versionNonce": 787992294, "isDeleted": false, "boundElements": [], "updated": 1760910920723, "link": null, "locked": false, "text": "Content Seen?", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Content Seen?", "autoResize": true, "lineHeight": 1.25 }, { "id": "MmO5yI92bRm5pxOy7R99_", "type": "text", "x": 3019.745780902862, "y": 864.3598762285094, "width": 161.2799072265625, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ac", "roundness": null, "seed": 846994234, "version": 16, "versionNonce": 1172063866, "isDeleted": false, "boundElements": [], "updated": 1760910928817, "link": null, "locked": false, "text": "PNG Downloader", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "PNG Downloader", "autoResize": true, "lineHeight": 1.25 }, { "id": "YvkOWCYwgeySf-Pen0XGX", "type": "text", "x": 3324.7375287684154, "y": 870.740733392526, "width": 142.27989196777344, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ad", "roundness": null, "seed": 1314341222, "version": 16, "versionNonce": 822667814, "isDeleted": false, "boundElements": [], "updated": 1760910935267, "link": null, "locked": false, "text": "Link Extractor", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Link Extractor", "autoResize": true, "lineHeight": 1.25 }, { "id": "Aq-pz4UjB7FT7jlXviVLC", "type": "text", "x": 3709.737951954799, "y": 850.2591737391156, "width": 118.97991943359375, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ae", "roundness": null, "seed": 2095594490, "version": 13, "versionNonce": 255546618, "isDeleted": false, "boundElements": [], "updated": 1760910943082, "link": null, "locked": false, "text": "Web Monitor", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Web Monitor", "autoResize": true, "lineHeight": 1.25 }, { "id": "ZHPztTU0v-97Rg70SykHQ", "type": "text", "x": 3349.3682986502404, "y": 1169.2193783497319, "width": 100.81991577148438, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "af", "roundness": null, "seed": 1962355430, "version": 12, "versionNonce": 947422886, "isDeleted": false, "boundElements": [], "updated": 1760910949315, "link": null, "locked": false, "text": "URL Filter", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "URL Filter", "autoResize": true, "lineHeight": 1.25 }, { "id": "eqXLI9nM5TEKJV94lvErV", "type": "text", "x": 3381.156869444032, "y": 1402.8645471482, "width": 101.93991088867188, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ag", "roundness": null, "seed": 1851078522, "version": 11, "versionNonce": 1659438842, "isDeleted": false, "boundElements": [], "updated": 1760910956302, "link": null, "locked": false, "text": "URL Seen?", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "URL Seen?", "autoResize": true, "lineHeight": 1.25 }, { "id": "ztTwAbqgutZL-cQ_yEPl9", "type": "ellipse", "x": 3315.3976730749714, "y": 333.7734192506952, "width": 176.0025549877805, "height": 148.37145978142428, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ah", "roundness": { "type": 2 }, "seed": 1593512058, "version": 50, "versionNonce": 412067878, "isDeleted": false, "boundElements": [ { "id": "DojGmf3EC0T5LGC_8oCqw", "type": "arrow" } ], "updated": 1760910974765, "link": null, "locked": false }, { "id": "-JgC9NxfZ7TpLmovB5sK1", "type": "ellipse", "x": 3320.0014780495258, "y": 1520.8649452078903, "width": 176.0025549877805, "height": 148.37145978142428, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ai", "roundness": { "type": 2 }, "seed": 1854368762, "version": 53, "versionNonce": 1306136122, "isDeleted": false, "boundElements": [ { "id": "QV7JTfuXAHsrO_e5f0qPv", "type": "arrow" } ], "updated": 1760910979349, "link": null, "locked": false }, { "id": "DojGmf3EC0T5LGC_8oCqw", "type": "arrow", "x": 3402.7955793603455, "y": 482.51682018805354, "width": 3.0664481966973653, "height": 99.54798404587348, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aj", "roundness": null, "seed": 1369405222, "version": 39, "versionNonce": 1442034534, "isDeleted": false, "boundElements": [], "updated": 1760910974766, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -3.0664481966973653, 99.54798404587348 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "ztTwAbqgutZL-cQ_yEPl9", "focus": -0.018754552236971662, "gap": 1 }, "endBinding": { "elementId": "eKWO_j7vSgr7VHYeJHm4p", "focus": 0.018713297990293122, "gap": 3.719411559336777 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "QV7JTfuXAHsrO_e5f0qPv", "type": "arrow", "x": 3417.4417955451086, "y": 1456.8538722717176, "width": 0, "height": 71.04076078331809, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ak", "roundness": null, "seed": 391669626, "version": 13, "versionNonce": 222154106, "isDeleted": false, "boundElements": [], "updated": 1760910979349, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 0, 71.04076078331809 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "VyNrta42IqLTjEbY2qqBH", "focus": 0.06290707950119685, "gap": 1 }, "endBinding": { "elementId": "-JgC9NxfZ7TpLmovB5sK1", "focus": 0.10726026110640953, "gap": 6.572737288103614 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "f3mOkcPphnKu6p3wWhT-1", "type": "text", "x": 3362.477158057143, "y": 1580.3548667597152, "width": 124.47990417480469, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "al", "roundness": null, "seed": 281955750, "version": 13, "versionNonce": 84788390, "isDeleted": false, "boundElements": [], "updated": 1760910985006, "link": null, "locked": false, "text": "URL Storage", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "URL Storage", "autoResize": true, "lineHeight": 1.25 }, { "id": "fxLQ6eUEVkti5zHbgFMKa", "type": "text", "x": 3354.930885271201, "y": 391.80450493535864, "width": 86.47994995117188, "height": 50, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "am", "roundness": null, "seed": 641112442, "version": 42, "versionNonce": 1607246758, "isDeleted": false, "boundElements": [], "updated": 1760910998872, "link": null, "locked": false, "text": "Content \nStorage", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Content \nStorage", "autoResize": true, "lineHeight": 1.25 }, { "id": "wmnlF0bpBpzeQNvOo0SlJ", "type": "text", "x": 327.314474811884, "y": 757.067748403121, "width": 184.77987670898438, "height": 100, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "an", "roundness": null, "seed": 2087915561, "version": 125, "versionNonce": 1782935113, "isDeleted": false, "boundElements": [], "updated": 1766974317521, "link": null, "locked": false, "text": "Core Entities\n- Text Data\n- URL metadata\n- Domain metadata", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Core Entities\n- Text Data\n- URL metadata\n- Domain metadata", "autoResize": true, "lineHeight": 1.25 }, { "id": "xKfUdstF1LISNHuQmFMKk", "type": "text", "x": 700.539277391779, "y": 750.8704724094739, "width": 241.2798309326172, "height": 75, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ao", "roundness": null, "seed": 1709091081, "version": 89, "versionNonce": 1817986569, "isDeleted": false, "boundElements": [], "updated": 1766974315670, "link": null, "locked": false, "text": "Interface\n- Input: set of seed urls\n- output: text data", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Interface\n- Input: set of seed urls\n- output: text data", "autoResize": true, "lineHeight": 1.25 }, { "id": "DuJK7DtdGfN-ieGRqnjD-", "type": "rectangle", "x": 144.74077833465645, "y": 1373.3942683106052, "width": 154.89453951066798, "height": 75.73900717267101, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ap", "roundness": { "type": 3 }, "seed": 1088571113, "version": 389, "versionNonce": 1197538633, "isDeleted": false, "boundElements": [ { "type": "text", "id": "YcZGsRwjzEpanLIYk1pUM" }, { "id": "MbVx2XJjScwwYvhmjY55V", "type": "arrow" }, { "id": "_2gJUk1MiZsww8QVourLW", "type": "arrow" }, { "id": "BPgimJxE3t7kdWM78QFNG", "type": "arrow" } ], "updated": 1766975885786, "link": null, "locked": false }, { "id": "YcZGsRwjzEpanLIYk1pUM", "type": "text", "x": 160.25810119057638, "y": 1386.2637718969409, "width": 123.85989379882812, "height": 50, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "apV", "roundness": null, "seed": 1402869353, "version": 57, "versionNonce": 650406953, "isDeleted": false, "boundElements": [], "updated": 1766975885786, "link": null, "locked": false, "text": "Frontier\nQueue (SQS)", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "DuJK7DtdGfN-ieGRqnjD-", "originalText": "Frontier Queue (SQS)", "autoResize": true, "lineHeight": 1.25 }, { "id": "ohc04Ye8617N8jh9eGROX", "type": "rectangle", "x": 461.9790856891549, "y": 1343.6530033793374, "width": 253.514883023622, "height": 165.95283499506215, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aq", "roundness": { "type": 3 }, "seed": 1336201929, "version": 360, "versionNonce": 1011650473, "isDeleted": false, "boundElements": [ { "type": "text", "id": "ZkmzKXY9DCxKw1DrxusSB" }, { "id": "MbVx2XJjScwwYvhmjY55V", "type": "arrow" }, { "id": "9hkg6RyQaKepqKk5Epj0g", "type": "arrow" }, { "id": "WtTjpYm3tx-Hu2mfO9YYY", "type": "arrow" }, { "id": "hTLOPcCuwljzOgp9HJe-B", "type": "arrow" }, { "id": "S39LzsEzligl3Q5aYBfH5", "type": "arrow" }, { "id": "7pAsBcsyYLbfNAakzXlCn", "type": "arrow" }, { "id": "BPgimJxE3t7kdWM78QFNG", "type": "arrow" }, { "id": "Gx9RwrTN_VzOzOSH3p_ex", "type": "arrow" }, { "id": "46e15j-GPTs71iVIm45bT", "type": "arrow" } ], "updated": 1766975885786, "link": null, "locked": false }, { "id": "ZkmzKXY9DCxKw1DrxusSB", "type": "text", "x": 468.9765937292862, "y": 1376.6294208768684, "width": 239.51986694335938, "height": 100, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aqV", "roundness": null, "seed": 807370441, "version": 110, "versionNonce": 2094065703, "isDeleted": false, "boundElements": [], "updated": 1766977612390, "link": null, "locked": false, "text": "Crawler\n- pull a url off the queue\n- fetch the webpage\n- hash the html", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "ohc04Ye8617N8jh9eGROX", "originalText": "Crawler\n- pull a url off the queue\n- fetch the webpage\n- hash the html", "autoResize": true, "lineHeight": 1.25 }, { "id": "93H4EPGrjxTov3tOqQZLB", "type": "ellipse", "x": 976.8813000559335, "y": 1166.5764260573894, "width": 192.65560901877467, "height": 164.56732208577813, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "as", "roundness": { "type": 2 }, "seed": 1443715465, "version": 461, "versionNonce": 319002249, "isDeleted": false, "boundElements": [ { "type": "text", "id": "ARcshGF_Fe2mohTGB-L9t" }, { "id": "9hkg6RyQaKepqKk5Epj0g", "type": "arrow" }, { "id": "bsTRJE9d0C88FUsfWFWIQ", "type": "arrow" } ], "updated": 1767024467447, "link": null, "locked": false }, { "id": "ARcshGF_Fe2mohTGB-L9t", "type": "text", "x": 1011.5550980113662, "y": 1211.1767523959963, "width": 123.07992553710938, "height": 75, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "asV", "roundness": null, "seed": 867695913, "version": 315, "versionNonce": 580278057, "isDeleted": false, "boundElements": [], "updated": 1767024467448, "link": null, "locked": false, "text": "S3 \n- raw HTML\n- Text Data", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "93H4EPGrjxTov3tOqQZLB", "originalText": "S3 \n- raw HTML\n- Text Data", "autoResize": true, "lineHeight": 1.25 }, { "id": "YA7uB3bofwLOQSmSsTa1J", "type": "ellipse", "x": 917.1525476224948, "y": 1561.0320850027933, "width": 155.6922160042843, "height": 156, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "at", "roundness": { "type": 2 }, "seed": 1885673801, "version": 390, "versionNonce": 1592286921, "isDeleted": false, "boundElements": [ { "type": "text", "id": "vtUy_3iuy-vdE9MXcLOVN" }, { "id": "WtTjpYm3tx-Hu2mfO9YYY", "type": "arrow" }, { "id": "UwLjqszTiH2bm91sB6E-j", "type": "arrow" } ], "updated": 1766975885786, "link": null, "locked": false }, { "id": "vtUy_3iuy-vdE9MXcLOVN", "type": "text", "x": 945.9931762004474, "y": 1588.8777560702424, "width": 97.91993713378906, "height": 100, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "atV", "roundness": null, "seed": 665465321, "version": 246, "versionNonce": 1776524713, "isDeleted": false, "boundElements": [], "updated": 1766975885786, "link": null, "locked": false, "text": "URL\nMetadata\n(DynamoD\nB)", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "YA7uB3bofwLOQSmSsTa1J", "originalText": "URL Metadata\n(DynamoDB)", "autoResize": true, "lineHeight": 1.25 }, { "id": "1UQK2OGULR9yEZ5-RdxOe", "type": "rectangle", "x": 475.51742413727106, "y": 1130.3931969488401, "width": 157.94147338800394, "height": 82.63961283725257, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "av", "roundness": { "type": 3 }, "seed": 628628553, "version": 157, "versionNonce": 1299515241, "isDeleted": false, "boundElements": [ { "type": "text", "id": "zs6z-6kQLfDQw-zZD4Qu_" }, { "id": "S39LzsEzligl3Q5aYBfH5", "type": "arrow" } ], "updated": 1766975885786, "link": null, "locked": false }, { "id": "zs6z-6kQLfDQw-zZD4Qu_", "type": "text", "x": 513.9582001989488, "y": 1159.2130033674664, "width": 81.05992126464844, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "avV", "roundness": null, "seed": 550229127, "version": 145, "versionNonce": 2115121737, "isDeleted": false, "boundElements": [], "updated": 1766975885786, "link": null, "locked": false, "text": "Webpage", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "1UQK2OGULR9yEZ5-RdxOe", "originalText": "Webpage", "autoResize": true, "lineHeight": 1.25 }, { "id": "bvdp9s0mCHzUshQfFTG-I", "type": "rectangle", "x": 224.22738052765112, "y": 1122.5359152210942, "width": 122.681386903992, "height": 93.3012211263822, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aw", "roundness": { "type": 3 }, "seed": 1151888391, "version": 259, "versionNonce": 1396634633, "isDeleted": false, "boundElements": [ { "type": "text", "id": "Vmu0ZDVDFKDbpQnkScgqv" }, { "id": "7pAsBcsyYLbfNAakzXlCn", "type": "arrow" } ], "updated": 1766975885786, "link": null, "locked": false }, { "id": "Vmu0ZDVDFKDbpQnkScgqv", "type": "text", "x": 265.22808527115103, "y": 1156.6865257842853, "width": 40.67997741699219, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "awV", "roundness": null, "seed": 1427555337, "version": 128, "versionNonce": 1496681193, "isDeleted": false, "boundElements": [], "updated": 1766975885786, "link": null, "locked": false, "text": "DNS", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "bvdp9s0mCHzUshQfFTG-I", "originalText": "DNS", "autoResize": true, "lineHeight": 1.25 }, { "id": "ylCRJZ2iUGDoev3scZWb0", "type": "rectangle", "x": 662.9314552391569, "y": 1882.6563672916618, "width": 223.17098398884036, "height": 53.76780638820401, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b03", "roundness": { "type": 3 }, "seed": 537507463, "version": 229, "versionNonce": 529177865, "isDeleted": false, "boundElements": [ { "id": "hTLOPcCuwljzOgp9HJe-B", "type": "arrow" }, { "id": "AYc5jJSQI4EXa6zis8GD8", "type": "arrow" } ], "updated": 1766977680502, "link": null, "locked": false }, { "id": "lxjW7UQLoPZF-kfe2NiTj", "type": "rectangle", "x": 1414.2884796536673, "y": 1985.7053634602344, "width": 209.21336454104846, "height": 161.8253756476788, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b04", "roundness": { "type": 3 }, "seed": 1856433769, "version": 262, "versionNonce": 81284713, "isDeleted": false, "boundElements": [ { "type": "text", "id": "E7ywVOm26e7jNjdeoDe50" }, { "id": "AYc5jJSQI4EXa6zis8GD8", "type": "arrow" }, { "id": "bsTRJE9d0C88FUsfWFWIQ", "type": "arrow" }, { "id": "_2gJUk1MiZsww8QVourLW", "type": "arrow" }, { "id": "UwLjqszTiH2bm91sB6E-j", "type": "arrow" } ], "updated": 1766975885786, "link": null, "locked": false }, { "id": "E7ywVOm26e7jNjdeoDe50", "type": "text", "x": 1427.0152180765353, "y": 2016.6180512840738, "width": 183.7598876953125, "height": 100, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b05", "roundness": null, "seed": 1676870503, "version": 258, "versionNonce": 393517385, "isDeleted": false, "boundElements": [], "updated": 1766975885786, "link": null, "locked": false, "text": "Parsing Worker\n- extract the text\n- save that text \n- extract url", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "lxjW7UQLoPZF-kfe2NiTj", "originalText": "Parsing Worker\n- extract the text\n- save that text \n- extract url", "autoResize": true, "lineHeight": 1.25 }, { "id": "jJAk7x1BnYJ2bzgpAa_NJ", "type": "text", "x": 712.0960691449474, "y": 1895.6293354667728, "width": 137.0398712158203, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b06", "roundness": null, "seed": 122590215, "version": 225, "versionNonce": 1613475529, "isDeleted": false, "boundElements": [], "updated": 1766977680502, "link": null, "locked": false, "text": "Parsing Queue", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Parsing Queue", "autoResize": true, "lineHeight": 1.25 }, { "id": "7b5EaQG9nISmN62A0nFNw", "type": "rectangle", "x": 235.44497570049873, "y": 1473.302799717212, "width": 69.90443693699126, "height": 35, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b07", "roundness": { "type": 3 }, "seed": 186373001, "version": 76, "versionNonce": 486362761, "isDeleted": false, "boundElements": [ { "type": "text", "id": "5eD0yHFBHlqEz0clzhjQX" } ], "updated": 1766975885786, "link": null, "locked": false }, { "id": "5eD0yHFBHlqEz0clzhjQX", "type": "text", "x": 249.48721339506858, "y": 1478.302799717212, "width": 41.81996154785156, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b08", "roundness": null, "seed": 1317463465, "version": 39, "versionNonce": 1061459305, "isDeleted": false, "boundElements": [], "updated": 1766975885786, "link": null, "locked": false, "text": "DLQ", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "7b5EaQG9nISmN62A0nFNw", "originalText": "DLQ", "autoResize": true, "lineHeight": 1.25 }, { "id": "MbVx2XJjScwwYvhmjY55V", "type": "arrow", "x": 302.8522368793371, "y": 1419.7558536360748, "width": 155.58203895667108, "height": 8.004901739599973, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b09", "roundness": { "type": 2 }, "seed": 949652937, "version": 186, "versionNonce": 765576263, "isDeleted": false, "boundElements": [], "updated": 1766975885786, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 155.58203895667108, 8.004901739599973 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "DuJK7DtdGfN-ieGRqnjD-", "focus": 0.10373619481225943, "gap": 3.216919034012676 }, "endBinding": { "elementId": "ohc04Ye8617N8jh9eGROX", "focus": -0.08754983739371301, "gap": 3.5448098531467167 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "9hkg6RyQaKepqKk5Epj0g", "type": "arrow", "x": 719.5374595005459, "y": 1386.2267145923377, "width": 254.61541973978296, "height": 124.71856275835421, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0A", "roundness": { "type": 2 }, "seed": 339311495, "version": 319, "versionNonce": 759291241, "isDeleted": false, "boundElements": [ { "type": "text", "id": "ncZ__pGtISY6wBoK1P2um" } ], "updated": 1767024467448, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 254.61541973978296, -124.71856275835421 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "ohc04Ye8617N8jh9eGROX", "focus": 0.1625014496561729, "gap": 4.861237235377757 }, "endBinding": { "elementId": "93H4EPGrjxTov3tOqQZLB", "focus": 0.27738219895696525, "gap": 3.814866761293126 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "ncZ__pGtISY6wBoK1P2um", "type": "text", "x": 347.1633179398641, "y": 2136.2297827941256, "width": 160.159912109375, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0AV", "roundness": null, "seed": 1658477289, "version": 16, "versionNonce": 1976756457, "isDeleted": false, "boundElements": [], "updated": 1766974930442, "link": null, "locked": false, "text": "Save that HTML", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "9hkg6RyQaKepqKk5Epj0g", "originalText": "Save that HTML", "autoResize": true, "lineHeight": 1.25 }, { "id": "WtTjpYm3tx-Hu2mfO9YYY", "type": "arrow", "x": 717.0865669574592, "y": 1498.9017686905263, "width": 199.87811126806764, "height": 112.2643056107213, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0B", "roundness": { "type": 2 }, "seed": 1048485767, "version": 327, "versionNonce": 67496135, "isDeleted": false, "boundElements": [ { "type": "text", "id": "_8lZ5k3weTh20NSJhHcwH" } ], "updated": 1766975885786, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 199.87811126806764, 112.2643056107213 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "ohc04Ye8617N8jh9eGROX", "focus": 0.0011855164674011286, "gap": 6.189019326440012 }, "endBinding": { "elementId": "YA7uB3bofwLOQSmSsTa1J", "focus": -0.13114043316241622, "gap": 4.99674976797172 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "_8lZ5k3weTh20NSJhHcwH", "type": "text", "x": 344.4561973135246, "y": 2257.0454148851873, "width": 193.25985717773438, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0BV", "roundness": null, "seed": 1348971591, "version": 19, "versionNonce": 1865969575, "isDeleted": false, "boundElements": [], "updated": 1766974924858, "link": null, "locked": false, "text": "Update URL status", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "WtTjpYm3tx-Hu2mfO9YYY", "originalText": "Update URL status", "autoResize": true, "lineHeight": 1.25 }, { "id": "hTLOPcCuwljzOgp9HJe-B", "type": "arrow", "x": 576.2854180374189, "y": 1511.1702767727218, "width": 85.71169178712557, "height": 372.24164138065476, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0C", "roundness": { "type": 2 }, "seed": 265021257, "version": 351, "versionNonce": 1344541673, "isDeleted": false, "boundElements": [], "updated": 1766977680502, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 85.71169178712557, 372.24164138065476 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "ohc04Ye8617N8jh9eGROX", "focus": 0.2189313837856253, "gap": 2.833633695081744 }, "endBinding": { "elementId": "ylCRJZ2iUGDoev3scZWb0", "focus": -0.9044153786013361, "gap": 5.765447621041643 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "AYc5jJSQI4EXa6zis8GD8", "type": "arrow", "x": 892.7787583834782, "y": 1918.6602442010274, "width": 519.690157145424, "height": 151.18028211041133, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0D", "roundness": { "type": 2 }, "seed": 1241049673, "version": 515, "versionNonce": 1105226567, "isDeleted": false, "boundElements": [], "updated": 1766977687085, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 519.690157145424, 151.18028211041133 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "ylCRJZ2iUGDoev3scZWb0", "focus": -0.4260357159198387, "gap": 6.676319155480996 }, "endBinding": { "elementId": "lxjW7UQLoPZF-kfe2NiTj", "focus": -0.3069998614872401, "gap": 1.8195641247650656 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "bsTRJE9d0C88FUsfWFWIQ", "type": "arrow", "x": 1516.805534467993, "y": 1983.1370078698392, "width": 336.92135806711326, "height": 715.9087693532151, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0E", "roundness": { "type": 2 }, "seed": 396547369, "version": 585, "versionNonce": 1683473287, "isDeleted": false, "boundElements": [ { "type": "text", "id": "xflqislVnfpy33362XRlN" } ], "updated": 1767024472409, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -110.94636029083608, -545.1871006311121 ], [ -336.92135806711326, -715.9087693532151 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "lxjW7UQLoPZF-kfe2NiTj", "focus": 0.12305751792682579, "gap": 2.5683555903951856 }, "endBinding": { "elementId": "93H4EPGrjxTov3tOqQZLB", "focus": -0.4012924301287706, "gap": 12.407001572182137 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "xflqislVnfpy33362XRlN", "type": "text", "x": 1289.7492422313562, "y": 1400.449907238727, "width": 232.21986389160156, "height": 75, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0EV", "roundness": null, "seed": 505149063, "version": 62, "versionNonce": 584991175, "isDeleted": false, "boundElements": [], "updated": 1766983038250, "link": null, "locked": false, "text": "1. Fetch the HTML from\nS3 Link\n2. Store the raw text", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "bsTRJE9d0C88FUsfWFWIQ", "originalText": "1. Fetch the HTML from S3 Link\n2. Store the raw text", "autoResize": true, "lineHeight": 1.25 }, { "id": "_2gJUk1MiZsww8QVourLW", "type": "arrow", "x": 1417.233452179909, "y": 2147.423833974439, "width": 1277.2216163442279, "height": 727.0147207521018, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0F", "roundness": { "type": 2 }, "seed": 1168618503, "version": 415, "versionNonce": 462123463, "isDeleted": false, "boundElements": [ { "type": "text", "id": "gbRbKwuBPkLaVMx1S5SUO" } ], "updated": 1766975885787, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -1115.7955688492566, -221.32692951296167 ], [ -1277.2216163442279, -727.0147207521018 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "lxjW7UQLoPZF-kfe2NiTj", "focus": -1.6782904823515186, "gap": 9.218963002555414 }, "endBinding": { "elementId": "DuJK7DtdGfN-ieGRqnjD-", "focus": 1.3019813227288737, "gap": 4.728942498975414 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "gbRbKwuBPkLaVMx1S5SUO", "type": "text", "x": -176.24351096684177, "y": 2723.573892629667, "width": 365.07977294921875, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0FV", "roundness": null, "seed": 177297417, "version": 38, "versionNonce": 314546505, "isDeleted": false, "boundElements": [], "updated": 1766975015045, "link": null, "locked": false, "text": "put extracted urls back on the queue", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "_2gJUk1MiZsww8QVourLW", "originalText": "put extracted urls back on the queue", "autoResize": true, "lineHeight": 1.25 }, { "id": "S39LzsEzligl3Q5aYBfH5", "type": "arrow", "x": 574.1358461967714, "y": 1340.1403635684674, "width": 10.216658843709013, "height": 126.54215674419675, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0G", "roundness": { "type": 2 }, "seed": 480153225, "version": 279, "versionNonce": 1697704967, "isDeleted": false, "boundElements": [], "updated": 1766975885787, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 10.216658843709013, -126.54215674419675 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "ohc04Ye8617N8jh9eGROX", "focus": -0.1617270689815185, "gap": 3.5126398108700414 }, "endBinding": { "elementId": "1UQK2OGULR9yEZ5-RdxOe", "focus": -0.4039282944547889, "gap": 1 }, "startArrowhead": "arrow", "endArrowhead": "arrow", "elbowed": false }, { "id": "7pAsBcsyYLbfNAakzXlCn", "type": "arrow", "x": 502.2502313144944, "y": 1335.7506704941961, "width": 158.24795169675141, "height": 112.17277489774324, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0H", "roundness": { "type": 2 }, "seed": 56130151, "version": 266, "versionNonce": 1726797383, "isDeleted": false, "boundElements": [], "updated": 1766975885787, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -158.24795169675141, -112.17277489774324 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "ohc04Ye8617N8jh9eGROX", "focus": 0.17111662950068382, "gap": 7.902332885141277 }, "endBinding": { "elementId": "bvdp9s0mCHzUshQfFTG-I", "focus": 0.14390932409378887, "gap": 12.644745611138143 }, "startArrowhead": "arrow", "endArrowhead": "arrow", "elbowed": false }, { "id": "BPgimJxE3t7kdWM78QFNG", "type": "arrow", "x": 576.1084348698233, "y": 1518.5998522206887, "width": 381.283221405548, "height": 258.86478058525654, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0I", "roundness": { "type": 2 }, "seed": 2066569289, "version": 328, "versionNonce": 31666311, "isDeleted": false, "boundElements": [ { "type": "text", "id": "MQMlIS9nVKqRHGTh9M3uE" } ], "updated": 1766975885787, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -221.71673592782088, 194.05827290912475 ], [ -381.283221405548, -64.80650767613179 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "ohc04Ye8617N8jh9eGROX", "focus": -0.41727065010246306, "gap": 8.994013846289135 }, "endBinding": { "elementId": "DuJK7DtdGfN-ieGRqnjD-", "focus": 0.5315828919884256, "gap": 4.6600690612806375 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "MQMlIS9nVKqRHGTh9M3uE", "type": "text", "x": -129.10433341152478, "y": 2440.043272322996, "width": 203.3998565673828, "height": 50, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0J", "roundness": null, "seed": 549042985, "version": 46, "versionNonce": 1374481257, "isDeleted": false, "boundElements": [], "updated": 1766975005674, "link": null, "locked": false, "text": "Retry on failure with\nexponential backoff", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "BPgimJxE3t7kdWM78QFNG", "originalText": "Retry on failure with exponential backoff", "autoResize": true, "lineHeight": 1.25 }, { "id": "oxYlmHDrR-pZyND8mjKIh", "type": "text", "x": 672.2160065635238, "y": 1949.5005853023526, "width": 131.01988220214844, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0K", "roundness": null, "seed": 1041446407, "version": 200, "versionNonce": 104656297, "isDeleted": false, "boundElements": [ { "id": "AYc5jJSQI4EXa6zis8GD8", "type": "arrow" } ], "updated": 1766977680502, "link": null, "locked": false, "text": "{url: , s3Link}", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "{url: , s3Link}", "autoResize": true, "lineHeight": 1.25 }, { "id": "daKrvhLHS6-7fVmIh98OD", "type": "text", "x": 1297.8213663434967, "y": 1205.5831443302347, "width": 195.4799041748047, "height": 100, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0L", "roundness": null, "seed": 627112649, "version": 124, "versionNonce": 1561717097, "isDeleted": false, "boundElements": [], "updated": 1766976918859, "link": null, "locked": false, "text": "Example Robots.txt\nUser-agent: *\nDisallow: /private/\nCrawl-delay: 10", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Example Robots.txt\nUser-agent: *\nDisallow: /private/\nCrawl-delay: 10", "autoResize": true, "lineHeight": 1.25 }, { "id": "VLPWzdgXZoAxC1CZtQpxU", "type": "text", "x": 254.5624179718602, "y": 1525.0833982903628, "width": 288.099853515625, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0M", "roundness": null, "seed": 918427145, "version": 64, "versionNonce": 947763625, "isDeleted": false, "boundElements": [], "updated": 1766975885786, "link": null, "locked": false, "text": "on 5th attempt move to DLQ", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "on 5th attempt move to DLQ", "autoResize": true, "lineHeight": 1.25 }, { "id": "pbQpRX2ylNovDkxcqutPI", "type": "text", "x": 114.66864756147527, "y": 1336.792313498268, "width": 215.5398406982422, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0N", "roundness": null, "seed": 674976967, "version": 86, "versionNonce": 1640668297, "isDeleted": false, "boundElements": [], "updated": 1766975885786, "link": null, "locked": false, "text": "Start with seed URLs", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Start with seed URLs", "autoResize": true, "lineHeight": 1.25 }, { "id": "NvZH57XydHxQm505V93rS", "type": "text", "x": 1087.0749926412705, "y": 1515.054041825923, "width": 169.1998748779297, "height": 175, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0O", "roundness": null, "seed": 1145353511, "version": 268, "versionNonce": 1496851305, "isDeleted": false, "boundElements": [ { "id": "UwLjqszTiH2bm91sB6E-j", "type": "arrow" } ], "updated": 1766975885786, "link": null, "locked": false, "text": "URL\n- id (PK)\n- url\n- s3Link to HTML\n- lastCrawlTime\n- Hash (GSI)\n- depth", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "URL\n- id (PK)\n- url\n- s3Link to HTML\n- lastCrawlTime\n- Hash (GSI)\n- depth", "autoResize": true, "lineHeight": 1.25 }, { "id": "oBcJVFx3BBgW-08hZQfc6", "type": "text", "x": 965.4705986906642, "y": 1721.739818606885, "width": 173.51986694335938, "height": 150, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0P", "roundness": null, "seed": 1030279751, "version": 215, "versionNonce": 277360711, "isDeleted": false, "boundElements": [ { "id": "UwLjqszTiH2bm91sB6E-j", "type": "arrow" } ], "updated": 1766977008191, "link": null, "locked": false, "text": "Domain\n- domain\n- lastCrawledTime\n- userAgent\n- disallow\n- crawlDelay", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Domain\n- domain\n- lastCrawledTime\n- userAgent\n- disallow\n- crawlDelay", "autoResize": true, "lineHeight": 1.25 }, { "id": "73s930ld02TSPXutgjFgD", "type": "ellipse", "x": 722.528645573891, "y": 1012.9599224376366, "width": 236.6505733853191, "height": 156.25588744268384, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0Q", "roundness": { "type": 2 }, "seed": 1032582249, "version": 135, "versionNonce": 318843049, "isDeleted": false, "boundElements": [ { "id": "Gx9RwrTN_VzOzOSH3p_ex", "type": "arrow" }, { "type": "text", "id": "XOQfHpIKpkKJ8LGrENZjR" } ], "updated": 1767024462046, "link": null, "locked": false }, { "id": "XOQfHpIKpkKJ8LGrENZjR", "type": "text", "x": 775.6553742867935, "y": 1053.3430673534567, "width": 130.0598907470703, "height": 75, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0QV", "roundness": null, "seed": 430260647, "version": 119, "versionNonce": 787206761, "isDeleted": false, "boundElements": [], "updated": 1767024462046, "link": null, "locked": false, "text": "Redis\n- rate limiting\n- dns caching", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "73s930ld02TSPXutgjFgD", "originalText": "Redis\n- rate limiting\n- dns caching", "autoResize": true, "lineHeight": 1.25 }, { "id": "Gx9RwrTN_VzOzOSH3p_ex", "type": "arrow", "x": 639.7377530914295, "y": 1340.662714529757, "width": 163.95891046282998, "height": 173.17840542423892, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0R", "roundness": { "type": 2 }, "seed": 733339849, "version": 298, "versionNonce": 403188617, "isDeleted": false, "boundElements": [], "updated": 1767024462046, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 163.95891046282998, -173.17840542423892 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "ohc04Ye8617N8jh9eGROX", "focus": -0.14826467102386567, "gap": 5.379645481912576 }, "endBinding": { "elementId": "73s930ld02TSPXutgjFgD", "focus": -0.18293218962513028, "gap": 2.1815230996160877 }, "startArrowhead": "arrow", "endArrowhead": "arrow", "elbowed": false }, { "id": "UwLjqszTiH2bm91sB6E-j", "type": "arrow", "x": 1466.6187512400293, "y": 1979.8685047105696, "width": 408.6565453927851, "height": 301.4170708210959, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0S", "roundness": { "type": 2 }, "seed": 769337415, "version": 440, "versionNonce": 1796158727, "isDeleted": false, "boundElements": [ { "type": "text", "id": "C9foNbyAcQKGFUThl7tYF" } ], "updated": 1766975885788, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -408.6565453927851, -301.4170708210959 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "lxjW7UQLoPZF-kfe2NiTj", "focus": 0.30487718952521947, "gap": 5.836858749664771 }, "endBinding": { "elementId": "NvZH57XydHxQm505V93rS", "focus": 0, "gap": 29.112786794026306 }, "startArrowhead": "arrow", "endArrowhead": "arrow", "elbowed": false }, { "id": "C9foNbyAcQKGFUThl7tYF", "type": "text", "x": 778.9732381568454, "y": 2537.6273062096825, "width": 217.3198699951172, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0T", "roundness": null, "seed": 1601139751, "version": 24, "versionNonce": 1681452839, "isDeleted": false, "boundElements": [], "updated": 1766975418607, "link": null, "locked": false, "text": "do we already have url", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "UwLjqszTiH2bm91sB6E-j", "originalText": "do we already have url", "autoResize": true, "lineHeight": 1.25 }, { "id": "KBN9hLbX5BDHFv4QXQXWO", "type": "ellipse", "x": 697.9075776512746, "y": 1627.2247143349664, "width": 182.0257954522813, "height": 127.76044498080228, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0U", "roundness": { "type": 2 }, "seed": 666100905, "version": 114, "versionNonce": 113207913, "isDeleted": false, "boundElements": [ { "type": "text", "id": "ZZW9e71kMIT1fO8441NqY" }, { "id": "46e15j-GPTs71iVIm45bT", "type": "arrow" } ], "updated": 1766975885786, "link": null, "locked": false }, { "id": "ZZW9e71kMIT1fO8441NqY", "type": "text", "x": 729.6646824703117, "y": 1665.9347983186994, "width": 118.79991149902344, "height": 50, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0V", "roundness": null, "seed": 770344135, "version": 117, "versionNonce": 102249801, "isDeleted": false, "boundElements": [], "updated": 1766975885786, "link": null, "locked": false, "text": "Redis\n- set (hash)", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "KBN9hLbX5BDHFv4QXQXWO", "originalText": "Redis\n- set (hash)", "autoResize": true, "lineHeight": 1.25 }, { "id": "46e15j-GPTs71iVIm45bT", "type": "arrow", "x": 650.9946871118841, "y": 1521.569257812671, "width": 81.04550671569814, "height": 114.51103593090465, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0W", "roundness": { "type": 2 }, "seed": 40261353, "version": 140, "versionNonce": 512548679, "isDeleted": false, "boundElements": [], "updated": 1766975885788, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 81.04550671569814, 114.51103593090465 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "ohc04Ye8617N8jh9eGROX", "focus": 0.026610217758486442, "gap": 11.96341943827133 }, "endBinding": { "elementId": "KBN9hLbX5BDHFv4QXQXWO", "focus": -0.1316683996992225, "gap": 4.527903651522741 }, "startArrowhead": "arrow", "endArrowhead": "arrow", "elbowed": false }, { "id": "W3GEpO_aPNDVJXt9ZV55R", "type": "text", "x": -52.52634018068943, "y": 2171.3230760405463, "width": 917.8353271484375, "height": 170.52220990433796, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0Z", "roundness": null, "seed": 1769257577, "version": 287, "versionNonce": 192324569, "isDeleted": false, "boundElements": [], "updated": 1772665076387, "link": null, "locked": false, "text": "1. How to handle dynamic content: server side rendering\n2. how to monitor health of the system\n3. how to handle large webpages (content-length)\n4. how to handle continual updates", "fontSize": 34.10444198086759, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "1. How to handle dynamic content: server side rendering\n2. how to monitor health of the system\n3. how to handle large webpages (content-length)\n4. how to handle continual updates", "autoResize": true, "lineHeight": 1.25 } ], "appState": { "gridSize": 20, "gridStep": 5, "gridModeEnabled": false, "viewBackgroundColor": "#ffffff" }, "files": {} }