Log Ingestion Service

{ "type": "excalidraw", "version": 2, "source": "https://marketplace.visualstudio.com/items?itemName=pomdtr.excalidraw-editor", "elements": [ { "id": "hUeuH-nfAiG1VV-CzuRJO", "type": "text", "x": -259.3774736157915, "y": -442.86947058105466, "width": 445.2480773925781, "height": 40, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "dotted", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0", "roundness": null, "seed": 1602006762, "version": 92, "versionNonce": 220731818, "isDeleted": false, "boundElements": [], "updated": 1773090223507, "link": null, "locked": false, "text": "Design a log ingestion system (i.e. Splunk, Datadog, etc.)\n", "fontSize": 16, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Design a log ingestion system (i.e. Splunk, Datadog, etc.)\n", "autoResize": true, "lineHeight": 1.25 }, { "id": "I0JtEQLmHBSyKgtVmSy8Y", "type": "text", "x": -1019.8066677517369, "y": -364.9933766412064, "width": 859.739501953125, "height": 925, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a1", "roundness": null, "seed": 1520124330, "version": 1150, "versionNonce": 602128391, "isDeleted": false, "boundElements": [], "updated": 1773096256813, "link": null, "locked": false, "text": "Functional Requ\n- system should be able to ingest logs from multiple applications\n- system should ensure a msg is only logged once every 10 sec on a per applicatoin basis\n- System should enable low latency log tailing via a dashboard\n\noos\n-alerts and visualization\n\nNon Functional Req\n- Scale: 50M logs per second\n- latency for log ingestion is <2s\n- Availability >> eventual consistency\n\nEntites\n- LogEvent (id, body,level)\n- Application\n\nAPI Design\n\nPOST /v1/logs->log_id\nHeaders->Auth: Bearer {Token}\nBody{\n message: string,\n log_level: DEBUG | INFO | WARN | ERROR\n}\n\n// SSE for subscribing to live updates for log tailing\n/log_update->{\n message: string,\n log_level: string,\n application_id: string,\n log_time: timestamp,\n log_id: UUID\n}\n\n\n", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Functional Requ\n- system should be able to ingest logs from multiple applications\n- system should ensure a msg is only logged once every 10 sec on a per applicatoin basis\n- System should enable low latency log tailing via a dashboard\n\noos\n-alerts and visualization\n\nNon Functional Req\n- Scale: 50M logs per second\n- latency for log ingestion is <2s\n- Availability >> eventual consistency\n\nEntites\n- LogEvent (id, body,level)\n- Application\n\nAPI Design\n\nPOST /v1/logs->log_id\nHeaders->Auth: Bearer {Token}\nBody{\n message: string,\n log_level: DEBUG | INFO | WARN | ERROR\n}\n\n// SSE for subscribing to live updates for log tailing\n/log_update->{\n message: string,\n log_level: string,\n application_id: string,\n log_time: timestamp,\n log_id: UUID\n}\n\n\n", "autoResize": true, "lineHeight": 1.25 }, { "id": "x67LMBeVpwc8_ACaMogPT", "type": "rectangle", "x": -131.36395941840362, "y": -221.05500858564938, "width": 105.98524305555554, "height": 143.41145833333348, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a2", "roundness": { "type": 3 }, "seed": 31030378, "version": 264, "versionNonce": 321279367, "isDeleted": false, "boundElements": [ { "type": "text", "id": "iTmASmqyE6ZvFy9QG07iX" }, { "id": "OY-2S9DIcHbjE2R6B156a", "type": "arrow" } ], "updated": 1773096444374, "link": null, "locked": false }, { "id": "iTmASmqyE6ZvFy9QG07iX", "type": "text", "x": -105.5113143920907, "y": -161.84927941898263, "width": 54.27995300292969, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a3", "roundness": null, "seed": 50487082, "version": 208, "versionNonce": 586157223, "isDeleted": false, "boundElements": [], "updated": 1773096444374, "link": null, "locked": false, "text": "Client", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "x67LMBeVpwc8_ACaMogPT", "originalText": "Client", "autoResize": true, "lineHeight": 1.25 }, { "id": "LsvHg_AckHR6fVPDm7ldK", "type": "rectangle", "x": 391.18291558159706, "y": -345.65570303009554, "width": 123.56336805555543, "height": 103.95833333333402, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a4", "roundness": { "type": 3 }, "seed": 444656106, "version": 314, "versionNonce": 1341244935, "isDeleted": false, "boundElements": [ { "type": "text", "id": "iqrcizHJi1khrrCjAmc83" }, { "id": "OY-2S9DIcHbjE2R6B156a", "type": "arrow" }, { "id": "VJOzGhqzSJpys3BdGmVyc", "type": "arrow" }, { "id": "TsxAZrhrLnHYrClzbalFe", "type": "arrow" }, { "id": "HKYSHoxHLfXzUgjGF2c2H", "type": "arrow" }, { "id": "K_9GeP6M4vygnp2InIJZm", "type": "arrow" } ], "updated": 1773097601482, "link": null, "locked": false }, { "id": "iqrcizHJi1khrrCjAmc83", "type": "text", "x": 406.6746368408201, "y": -318.6765363634285, "width": 92.57992553710938, "height": 50, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a5", "roundness": null, "seed": 1063263402, "version": 295, "versionNonce": 246045991, "isDeleted": false, "boundElements": [], "updated": 1773096864184, "link": null, "locked": false, "text": "Ingestion\nSvc", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "LsvHg_AckHR6fVPDm7ldK", "originalText": "Ingestion Svc", "autoResize": true, "lineHeight": 1.25 }, { "id": "hJtN4XntvdOXL9IKqWEzr", "type": "rectangle", "x": -142.7840983072923, "y": 227.06999141436245, "width": 121.79253472222226, "height": 81.00694444444481, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a8", "roundness": { "type": 3 }, "seed": 806331626, "version": 278, "versionNonce": 1049420585, "isDeleted": false, "boundElements": [ { "type": "text", "id": "UZfHLFIrOJV8t_2_DHMs5" }, { "id": "ZbOhO7Lv_YhxCCxf0hv0O", "type": "arrow" } ], "updated": 1773096868877, "link": null, "locked": false }, { "id": "UZfHLFIrOJV8t_2_DHMs5", "type": "text", "x": -134.02779218885695, "y": 242.57346363658485, "width": 104.27992248535156, "height": 50, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a9", "roundness": null, "seed": 1230993322, "version": 262, "versionNonce": 2134414857, "isDeleted": false, "boundElements": [], "updated": 1773096868877, "link": null, "locked": false, "text": "Dashboard\nClient A", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "hJtN4XntvdOXL9IKqWEzr", "originalText": "Dashboard Client A", "autoResize": true, "lineHeight": 1.25 }, { "id": "TSsgrNZWM82-QJgalv47k", "type": "rectangle", "x": 153.73413085937443, "y": -314.1834808078602, "width": 108.30729166666663, "height": 762.6553819444448, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aA", "roundness": { "type": 3 }, "seed": 916910698, "version": 414, "versionNonce": 1170211113, "isDeleted": false, "boundElements": [ { "type": "text", "id": "XfsAK4kIXCdeG30sQ7Oqx" }, { "id": "ZbOhO7Lv_YhxCCxf0hv0O", "type": "arrow" }, { "id": "OY-2S9DIcHbjE2R6B156a", "type": "arrow" }, { "id": "TsxAZrhrLnHYrClzbalFe", "type": "arrow" }, { "id": "JOnXb_I4BryP-EYnZs7B5", "type": "arrow" }, { "id": "HKYSHoxHLfXzUgjGF2c2H", "type": "arrow" } ], "updated": 1773097556553, "link": null, "locked": false }, { "id": "XfsAK4kIXCdeG30sQ7Oqx", "type": "text", "x": 165.95906575520775, "y": -7.855789835637779, "width": 83.857421875, "height": 150, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aB", "roundness": null, "seed": 1789990186, "version": 327, "versionNonce": 1286520297, "isDeleted": false, "boundElements": [], "updated": 1773096719927, "link": null, "locked": false, "text": "API\nGateways \nLoad\nBalancing\nRate\nLimiting", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "TSsgrNZWM82-QJgalv47k", "originalText": "API Gateways \nLoad Balancing\nRate Limiting", "autoResize": true, "lineHeight": 1.25 }, { "id": "iM9qYcTvQUdddpMOA344O", "type": "rectangle", "x": 430.4407280815965, "y": 305.6706858588077, "width": 122.3654513888888, "height": 93.50260416666652, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aC", "roundness": { "type": 3 }, "seed": 1189666794, "version": 266, "versionNonce": 248896807, "isDeleted": false, "boundElements": [ { "type": "text", "id": "nV1jzAMTBGD2eeLIxz7LZ" }, { "id": "JOnXb_I4BryP-EYnZs7B5", "type": "arrow" }, { "id": "dj-AYUrLOI0YBLkkIeW3n", "type": "arrow" } ], "updated": 1773096834443, "link": null, "locked": false }, { "id": "nV1jzAMTBGD2eeLIxz7LZ", "type": "text", "x": 439.1734949747713, "y": 327.42198794214096, "width": 104.89991760253906, "height": 50, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aD", "roundness": null, "seed": 1197036202, "version": 257, "versionNonce": 2056101959, "isDeleted": false, "boundElements": [], "updated": 1773096834443, "link": null, "locked": false, "text": "Connection\nServers", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "iM9qYcTvQUdddpMOA344O", "originalText": "Connection Servers", "autoResize": true, "lineHeight": 1.25 }, { "id": "ZbOhO7Lv_YhxCCxf0hv0O", "type": "arrow", "x": -20.160998263104176, "y": 270.49450852808513, "width": 170.37576205742553, "height": 7.225626503978333, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aE", "roundness": { "type": 2 }, "seed": 783117674, "version": 566, "versionNonce": 1504373993, "isDeleted": false, "boundElements": [], "updated": 1773096868877, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 170.37576205742553, -7.225626503978333 ] ], "startBinding": { "elementId": "hJtN4XntvdOXL9IKqWEzr", "focus": 0.13100901221652408, "gap": 1 }, "endBinding": { "elementId": "TSsgrNZWM82-QJgalv47k", "focus": -0.5055595280682548, "gap": 19.47789360636972 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false, "moveMidPointsWithElement": false, "lastCommittedPoint": null }, { "id": "OY-2S9DIcHbjE2R6B156a", "type": "arrow", "x": -24.78251581515886, "y": -134.87347131357234, "width": 174.70078512035363, "height": 42.43215574409129, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aG", "roundness": { "type": 2 }, "seed": 634165994, "version": 571, "versionNonce": 1345226057, "isDeleted": false, "boundElements": [], "updated": 1773096719927, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 174.70078512035363, 42.43215574409129 ] ], "startBinding": { "elementId": "x67LMBeVpwc8_ACaMogPT", "focus": 0.018288612021662386, "gap": 1 }, "endBinding": { "elementId": "TSsgrNZWM82-QJgalv47k", "focus": 0.36877365767741177, "gap": 20.82031809497073 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false, "moveMidPointsWithElement": false, "lastCommittedPoint": null }, { "id": "KRT-AjIFGFbPqiEmTZ4sw", "type": "rectangle", "x": 589.8391655815971, "y": -370.78070303009554, "width": 206.20052083333346, "height": 96.4296875, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aH", "roundness": { "type": 3 }, "seed": 671985066, "version": 270, "versionNonce": 857111657, "isDeleted": false, "boundElements": [ { "type": "text", "id": "KG9OM23DURkZW1dv37FWj" }, { "id": "VJOzGhqzSJpys3BdGmVyc", "type": "arrow" }, { "id": "lyBm6ukOkzybzvBfUiAmQ", "type": "arrow" }, { "id": "2BVFpy1jIUZRpxhi5oQQb", "type": "arrow" } ], "updated": 1773096849794, "link": null, "locked": false }, { "id": "VJOzGhqzSJpys3BdGmVyc", "type": "arrow", "x": 515.5105323961092, "y": -285.9512758173486, "width": 68.95548119928822, "height": 48.30129210457045, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aI", "roundness": { "type": 2 }, "seed": 1310615658, "version": 667, "versionNonce": 194127017, "isDeleted": false, "boundElements": [], "updated": 1773096859194, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 68.95548119928822, -48.30129210457045 ] ], "startBinding": { "elementId": "LsvHg_AckHR6fVPDm7ldK", "focus": 0.5409363033918496, "gap": 1 }, "endBinding": { "elementId": "KRT-AjIFGFbPqiEmTZ4sw", "focus": 0.728342784837217, "gap": 6 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false, "moveMidPointsWithElement": false, "lastCommittedPoint": null }, { "id": "KG9OM23DURkZW1dv37FWj", "type": "text", "x": 615.1994891696505, "y": -335.06585928009554, "width": 155.47987365722656, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aJ", "roundness": null, "seed": 1338624810, "version": 265, "versionNonce": 1274314729, "isDeleted": false, "boundElements": [], "updated": 1773096849796, "link": null, "locked": false, "text": "Ingestion Kafka", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "KRT-AjIFGFbPqiEmTZ4sw", "originalText": "Ingestion Kafka", "autoResize": true, "lineHeight": 1.25 }, { "id": "c8znVkF9RRlp1jqrpW6fs", "type": "rectangle", "x": 906.4797905815973, "y": -274.5133419189606, "width": 162.84114583333346, "height": 67.72222222222216, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aK", "roundness": { "type": 3 }, "seed": 863688170, "version": 386, "versionNonce": 672018023, "isDeleted": false, "boundElements": [ { "type": "text", "id": "KCxnn2XEN_3z3C1NQPkdm" }, { "id": "lyBm6ukOkzybzvBfUiAmQ", "type": "arrow" }, { "id": "SRdK4lrJfWevvltL8UJ84", "type": "arrow" } ], "updated": 1773096846311, "link": null, "locked": false }, { "id": "KCxnn2XEN_3z3C1NQPkdm", "type": "text", "x": 948.4903903537327, "y": -253.1522308078495, "width": 78.8199462890625, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aL", "roundness": null, "seed": 2050787498, "version": 368, "versionNonce": 53070215, "isDeleted": false, "boundElements": [], "updated": 1773096846311, "link": null, "locked": false, "text": "Log SVC", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "c8znVkF9RRlp1jqrpW6fs", "originalText": "Log SVC", "autoResize": true, "lineHeight": 1.25 }, { "id": "SRdK4lrJfWevvltL8UJ84", "type": "arrow", "x": 1016.7647566115708, "y": -204.30186091781732, "width": 251.99133319924783, "height": 65.22212891832706, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aM", "roundness": { "type": 2 }, "seed": 1270473578, "version": 1077, "versionNonce": 795628487, "isDeleted": false, "boundElements": [], "updated": 1773096846312, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 251.99133319924783, 65.22212891832706 ] ], "startBinding": { "elementId": "c8znVkF9RRlp1jqrpW6fs", "focus": 0.525740267516384, "gap": 6.000000000000455 }, "endBinding": { "elementId": "XsJD6uG63vfG4QSojOR3L", "focus": 0.5039970879221856, "gap": 5.999896358514005 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false, "moveMidPointsWithElement": false, "lastCommittedPoint": null }, { "id": "lyBm6ukOkzybzvBfUiAmQ", "type": "arrow", "x": 796.9575583590682, "y": -327.96593986983487, "width": 107.50861894031209, "height": 68.27785632734304, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aN", "roundness": { "type": 2 }, "seed": 574481962, "version": 744, "versionNonce": 84882985, "isDeleted": false, "boundElements": [], "updated": 1773096849795, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 107.50861894031209, 68.27785632734304 ] ], "startBinding": { "elementId": "KRT-AjIFGFbPqiEmTZ4sw", "focus": -0.6299430463733324, "gap": 1.0153086878465274 }, "endBinding": { "elementId": "c8znVkF9RRlp1jqrpW6fs", "focus": -0.3955958289700644, "gap": 2.218091125099477 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false, "moveMidPointsWithElement": false, "lastCommittedPoint": null }, { "id": "XsJD6uG63vfG4QSojOR3L", "type": "ellipse", "x": 1219.633436414931, "y": -168.56195303005552, "width": 292.23437500000006, "height": 297, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aT", "roundness": { "type": 2 }, "seed": 1309261482, "version": 453, "versionNonce": 786380839, "isDeleted": false, "boundElements": [ { "type": "text", "id": "Nhtq4pbww_d11H3z-d622" }, { "id": "SRdK4lrJfWevvltL8UJ84", "type": "arrow" }, { "id": "MqtHtuCLDGKH1sISE9eGE", "type": "arrow" }, { "id": "oUPmxk_szY3Il2HIQCNxZ", "type": "arrow" } ], "updated": 1773096829144, "link": null, "locked": false }, { "id": "Nhtq4pbww_d11H3z-d622", "type": "text", "x": 1292.3702256329427, "y": -95.06731003625782, "width": 147.11988830566406, "height": 150, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aV", "roundness": null, "seed": 11901994, "version": 565, "versionNonce": 649304903, "isDeleted": false, "boundElements": [], "updated": 1773096829144, "link": null, "locked": false, "text": "Dynamo DB\npartition_key -\napp_id\nsort_key =\ntimestamp_id\nlogDB", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "XsJD6uG63vfG4QSojOR3L", "originalText": "Dynamo DB\npartition_key - app_id\nsort_key = timestamp_id\nlogDB", "autoResize": true, "lineHeight": 1.25 }, { "id": "trAbHmLqKGInDScN4TdFq", "type": "rectangle", "x": 901.759179009331, "y": -493.7099565023175, "width": 184.70052083333346, "height": 73.43315972222217, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aX", "roundness": { "type": 3 }, "seed": 941661353, "version": 274, "versionNonce": 975886567, "isDeleted": false, "boundElements": [ { "type": "text", "id": "7zRj6XTfUuL4yJXxgyBVg" }, { "id": "2BVFpy1jIUZRpxhi5oQQb", "type": "arrow" }, { "id": "F1lD8owrgVM-2PtxNoLT6", "type": "arrow" } ], "updated": 1773096856844, "link": null, "locked": false }, { "id": "7zRj6XTfUuL4yJXxgyBVg", "type": "text", "x": 939.4994845920133, "y": -469.4933766412064, "width": 109.21990966796875, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aY", "roundness": null, "seed": 1360600969, "version": 266, "versionNonce": 827139079, "isDeleted": false, "boundElements": [], "updated": 1773096856844, "link": null, "locked": false, "text": "Search SVC", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "trAbHmLqKGInDScN4TdFq", "originalText": "Search SVC", "autoResize": true, "lineHeight": 1.25 }, { "id": "2BVFpy1jIUZRpxhi5oQQb", "type": "arrow", "x": 798.5942592829093, "y": -360.83640955873926, "width": 108.49389204030797, "height": 59.38697321979396, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aZ", "roundness": { "type": 2 }, "seed": 1391711687, "version": 417, "versionNonce": 540609319, "isDeleted": false, "boundElements": [], "updated": 1773096856844, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 108.49389204030797, -59.38697321979396 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "KRT-AjIFGFbPqiEmTZ4sw", "focus": 0.18789264906538505, "gap": 5.08932528376639 }, "endBinding": { "elementId": "trAbHmLqKGInDScN4TdFq", "focus": 0.1237597158639658, "gap": 4.177772262579515 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "BR8wllDTtvBI2_NgcYE4Y", "type": "diamond", "x": 1187.1250644259976, "y": -579.7121266412064, "width": 195.765625, "height": 220, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aa", "roundness": { "type": 2 }, "seed": 1168892423, "version": 193, "versionNonce": 276751145, "isDeleted": false, "boundElements": [ { "type": "text", "id": "qOmHx30FZ6SCstGP8F2s6" }, { "id": "F1lD8owrgVM-2PtxNoLT6", "type": "arrow" } ], "updated": 1773096619410, "link": null, "locked": false }, { "id": "qOmHx30FZ6SCstGP8F2s6", "type": "text", "x": 1251.5464969211148, "y": -494.7121266412064, "width": 67.03994750976562, "height": 50, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ab", "roundness": null, "seed": 442269289, "version": 159, "versionNonce": 1122670089, "isDeleted": false, "boundElements": [], "updated": 1773096619410, "link": null, "locked": false, "text": "Elastic\nSearch", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "BR8wllDTtvBI2_NgcYE4Y", "originalText": "Elastic Search", "autoResize": true, "lineHeight": 1.25 }, { "id": "F1lD8owrgVM-2PtxNoLT6", "type": "arrow", "x": 1090.2309986008938, "y": -467.5273523999367, "width": 101.7872922127558, "height": 1.6382175145723181, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ac", "roundness": { "type": 2 }, "seed": 566209993, "version": 431, "versionNonce": 324734535, "isDeleted": false, "boundElements": [], "updated": 1773096856844, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 101.7872922127558, 1.6382175145723181 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "trAbHmLqKGInDScN4TdFq", "focus": -0.3174703408387075, "gap": 4.095052083333144 }, "endBinding": { "elementId": "BR8wllDTtvBI2_NgcYE4Y", "focus": -0.0478342453811768, "gap": 1.5621374639636687 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "sHqep_sAyjUmL9yRqgZXP", "type": "text", "x": 1410.4375644259976, "y": -493.1730641412064, "width": 433.59967041015625, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ad", "roundness": null, "seed": 691859143, "version": 177, "versionNonce": 323656615, "isDeleted": false, "boundElements": [], "updated": 1773097533326, "link": null, "locked": false, "text": "Used for full text search on logs dashboard", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Used for full text search on logs dashboard", "autoResize": true, "lineHeight": 1.25 }, { "id": "TsxAZrhrLnHYrClzbalFe", "type": "arrow", "x": 263.2357868036674, "y": -181.01140204104567, "width": 129.10131937635964, "height": 73.82682539019268, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ae", "roundness": { "type": 2 }, "seed": 1564903337, "version": 86, "versionNonce": 1293526921, "isDeleted": false, "boundElements": [], "updated": 1773096859195, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 129.10131937635964, -73.82682539019268 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "TSsgrNZWM82-QJgalv47k", "focus": -0.5252793489804409, "gap": 7.614891899956547 }, "endBinding": { "elementId": "LsvHg_AckHR6fVPDm7ldK", "focus": -0.0476967446459152, "gap": 1 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "JOnXb_I4BryP-EYnZs7B5", "type": "arrow", "x": 426.3163742041824, "y": 357.8039756249644, "width": 163.76645640304525, "height": 21.861261318067875, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "af", "roundness": { "type": 2 }, "seed": 618004263, "version": 203, "versionNonce": 480247655, "isDeleted": false, "boundElements": [], "updated": 1773096834443, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -163.76645640304525, -21.861261318067875 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "iM9qYcTvQUdddpMOA344O", "focus": -0.25819822200764325, "gap": 5.2375386555988825 }, "endBinding": { "elementId": "TSsgrNZWM82-QJgalv47k", "focus": 0.6734590347223686, "gap": 3.4352043999565467 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "rX7IbUIPOM_p9G8aEeTg1", "type": "text", "x": 302.5235019259976, "y": 280.4753733587936, "width": 114.83993530273438, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ag", "roundness": null, "seed": 930221577, "version": 136, "versionNonce": 369322505, "isDeleted": false, "boundElements": [], "updated": 1773096798793, "link": null, "locked": false, "text": "SSE Events", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "SSE Events", "autoResize": true, "lineHeight": 1.25 }, { "id": "MqtHtuCLDGKH1sISE9eGE", "type": "arrow", "x": 1346.8043327269213, "y": 131.43308666927857, "width": 15.423246212280901, "height": 91.18820899427033, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ah", "roundness": { "type": 2 }, "seed": 1198430183, "version": 317, "versionNonce": 592750887, "isDeleted": false, "boundElements": [], "updated": 1773096830742, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -15.423246212280901, 91.18820899427033 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "XsJD6uG63vfG4QSojOR3L", "focus": -0.04429198736947638, "gap": 4.2127325417538115 }, "endBinding": { "elementId": "08P80yIAtfgKch3NYkDEC", "focus": 0.036645013545908, "gap": 2.6523358798665413 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "Tno272EZXoY2P_0kl6vWO", "type": "rectangle", "x": 1971.4766269259976, "y": -84.0246266412064, "width": 205.984375, "height": 104.4609375, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ai", "roundness": { "type": 3 }, "seed": 989634569, "version": 217, "versionNonce": 1490023849, "isDeleted": false, "boundElements": [ { "type": "text", "id": "f987Sw3ivH4ndwxhCAv7S" }, { "id": "oUPmxk_szY3Il2HIQCNxZ", "type": "arrow" }, { "id": "caHb89b0Iad5l87xaVodW", "type": "arrow" } ], "updated": 1773097475628, "link": null, "locked": false }, { "id": "f987Sw3ivH4ndwxhCAv7S", "type": "text", "x": 1992.208880954318, "y": -44.294157891206396, "width": 164.51986694335938, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aj", "roundness": null, "seed": 1518650951, "version": 187, "versionNonce": 880105609, "isDeleted": false, "boundElements": [], "updated": 1773097475628, "link": null, "locked": false, "text": "Snapshot service", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "Tno272EZXoY2P_0kl6vWO", "originalText": "Snapshot service", "autoResize": true, "lineHeight": 1.25 }, { "id": "oUPmxk_szY3Il2HIQCNxZ", "type": "arrow", "x": 1513.2668616070282, "y": -50.671706716908886, "width": 457.3196045128245, "height": 8.756868230915593, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ak", "roundness": { "type": 2 }, "seed": 111238089, "version": 286, "versionNonce": 2133548905, "isDeleted": false, "boundElements": [], "updated": 1773097475628, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 457.3196045128245, 8.756868230915593 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "XsJD6uG63vfG4QSojOR3L", "focus": -0.22115807995173084, "gap": 4.445192077471441 }, "endBinding": { "elementId": "Tno272EZXoY2P_0kl6vWO", "focus": 0.15018797077550963, "gap": 1 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "v64G9AxE0b9hIseOyWjiV", "type": "ellipse", "x": 2012.3594394259976, "y": 278.0691233587936, "width": 223.4140625, "height": 196.8203125, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "al", "roundness": { "type": 2 }, "seed": 1454641065, "version": 98, "versionNonce": 92699465, "isDeleted": false, "boundElements": [ { "type": "text", "id": "klFM0AQvyUQj9jGsvLzgo" }, { "id": "caHb89b0Iad5l87xaVodW", "type": "arrow" } ], "updated": 1773097473412, "link": null, "locked": false }, { "id": "klFM0AQvyUQj9jGsvLzgo", "type": "text", "x": 2087.0877040267137, "y": 338.8927907867909, "width": 73.97993469238281, "height": 75, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "alV", "roundness": null, "seed": 1186996295, "version": 84, "versionNonce": 1066019369, "isDeleted": false, "boundElements": [], "updated": 1773097473412, "link": null, "locked": false, "text": "S3\nArchival\nGlacier", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "v64G9AxE0b9hIseOyWjiV", "originalText": "S3\nArchival\nGlacier", "autoResize": true, "lineHeight": 1.25 }, { "id": "caHb89b0Iad5l87xaVodW", "type": "arrow", "x": 2121.0150361474466, "y": 22.51416711060095, "width": 11.638198649055994, "height": 253.0418722008804, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "an", "roundness": { "type": 2 }, "seed": 579581927, "version": 249, "versionNonce": 1463650889, "isDeleted": false, "boundElements": [], "updated": 1773097475628, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 11.638198649055994, 253.0418722008804 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "Tno272EZXoY2P_0kl6vWO", "focus": -0.4179350623875721, "gap": 4.453125 }, "endBinding": { "elementId": "v64G9AxE0b9hIseOyWjiV", "focus": 0.11377516672541763, "gap": 2.7980592742653942 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "08P80yIAtfgKch3NYkDEC", "type": "rectangle", "x": 1215.9766269259976, "y": 223.7253733587936, "width": 208.140625, "height": 92.4140625, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ao", "roundness": { "type": 3 }, "seed": 772120103, "version": 177, "versionNonce": 989198055, "isDeleted": false, "boundElements": [ { "type": "text", "id": "7jS27V6jXcPHB1GHrNwHW" }, { "id": "MqtHtuCLDGKH1sISE9eGE", "type": "arrow" }, { "id": "DzOs26Da5CbHXvyc8rlN5", "type": "arrow" } ], "updated": 1773096830741, "link": null, "locked": false }, { "id": "7jS27V6jXcPHB1GHrNwHW", "type": "text", "x": 1284.8669696384, "y": 257.4324046087936, "width": 70.35993957519531, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ap", "roundness": null, "seed": 1361912935, "version": 118, "versionNonce": 1728348679, "isDeleted": false, "boundElements": [], "updated": 1773096830741, "link": null, "locked": false, "text": "Lambda", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "08P80yIAtfgKch3NYkDEC", "originalText": "Lambda", "autoResize": true, "lineHeight": 1.25 }, { "id": "Oj2_KuSMw9mZNfqPps0VS", "type": "rectangle", "x": 1019.9766269259976, "y": 408.5691233587936, "width": 437.7890625, "height": 124.2890625, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aq", "roundness": { "type": 3 }, "seed": 1919950759, "version": 105, "versionNonce": 556948487, "isDeleted": false, "boundElements": [ { "type": "text", "id": "niAVg_zfwlAVEQvY10Z2g" }, { "id": "PEpE7iCnKy0DA5S9eT2ir", "type": "arrow" }, { "id": "DzOs26Da5CbHXvyc8rlN5", "type": "arrow" } ], "updated": 1773096831777, "link": null, "locked": false }, { "id": "niAVg_zfwlAVEQvY10Z2g", "type": "text", "x": 1116.5612445407437, "y": 458.2136546087936, "width": 244.6198272705078, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ar", "roundness": null, "seed": 654620201, "version": 97, "versionNonce": 1320633127, "isDeleted": false, "boundElements": [], "updated": 1773096831777, "link": null, "locked": false, "text": "Kafka for live tailing logs", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "Oj2_KuSMw9mZNfqPps0VS", "originalText": "Kafka for live tailing logs", "autoResize": true, "lineHeight": 1.25 }, { "id": "u7swYVhnZBzPpTYS7m_RI", "type": "rectangle", "x": 700.1875644259976, "y": 299.7253733587936, "width": 94.2578125, "height": 96.3046875, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "as", "roundness": { "type": 3 }, "seed": 1102151273, "version": 87, "versionNonce": 1111827785, "isDeleted": false, "boundElements": [ { "id": "PEpE7iCnKy0DA5S9eT2ir", "type": "arrow" }, { "id": "dj-AYUrLOI0YBLkkIeW3n", "type": "arrow" }, { "type": "text", "id": "DQCiOz8Xl5oHjC6QxrzvR" } ], "updated": 1773096796877, "link": null, "locked": false }, { "id": "DQCiOz8Xl5oHjC6QxrzvR", "type": "text", "x": 706.4465060763882, "y": 322.8777171087936, "width": 81.73992919921875, "height": 50, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "asV", "roundness": null, "seed": 1571106407, "version": 60, "versionNonce": 48469033, "isDeleted": false, "boundElements": [], "updated": 1773096796877, "link": null, "locked": false, "text": "Redis \nPub/Sub", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "u7swYVhnZBzPpTYS7m_RI", "originalText": "Redis \nPub/Sub", "autoResize": true, "lineHeight": 1.25 }, { "id": "PEpE7iCnKy0DA5S9eT2ir", "type": "arrow", "x": 1040.7653816024997, "y": 409.0460084248258, "width": 241.6250194385559, "height": 59.26101373828152, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "at", "roundness": { "type": 2 }, "seed": 1086489159, "version": 99, "versionNonce": 1163019847, "isDeleted": false, "boundElements": [], "updated": 1773096831778, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -241.6250194385559, -59.26101373828152 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "Oj2_KuSMw9mZNfqPps0VS", "focus": 0.11291733704511658, "gap": 1.1640625 }, "endBinding": { "elementId": "u7swYVhnZBzPpTYS7m_RI", "focus": -0.18091046535012575, "gap": 6.703125 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "dj-AYUrLOI0YBLkkIeW3n", "type": "arrow", "x": 698.4598114913565, "y": 355.40029407168663, "width": 140.9821147424309, "height": 3.5491382649219076, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "au", "roundness": { "type": 2 }, "seed": 785216263, "version": 177, "versionNonce": 123304583, "isDeleted": false, "boundElements": [], "updated": 1773096834443, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -140.9821147424309, -3.5491382649219076 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "u7swYVhnZBzPpTYS7m_RI", "focus": -0.17891163629659296, "gap": 2.46875 }, "endBinding": { "elementId": "iM9qYcTvQUdddpMOA344O", "focus": -0.04484239753532791, "gap": 5.938927544487683 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "DzOs26Da5CbHXvyc8rlN5", "type": "arrow", "x": 1318.859780035119, "y": 319.3530195084747, "width": 64.73307724251868, "height": 83.64844766761126, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "av", "roundness": { "type": 2 }, "seed": 2108608329, "version": 83, "versionNonce": 870311271, "isDeleted": false, "boundElements": [], "updated": 1773096831778, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -64.73307724251868, 83.64844766761126 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "08P80yIAtfgKch3NYkDEC", "focus": -0.2649902504535206, "gap": 7.4140625 }, "endBinding": { "elementId": "Oj2_KuSMw9mZNfqPps0VS", "focus": -0.1391740491350055, "gap": 17.4453125 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "m1CIcpi1kcUqXUhvo7HPg", "type": "text", "x": 601.0547519259976, "y": -241.7277516412064, "width": 299.27984619140625, "height": 100, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aw", "roundness": null, "seed": 2099139687, "version": 115, "versionNonce": 616031399, "isDeleted": false, "boundElements": [], "updated": 1773096921689, "link": null, "locked": false, "text": "Ingestion Service\nattaches timestamp to log to \navoid client dependence on\nordering of messages", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Ingestion Service\nattaches timestamp to log to \navoid client dependence on\nordering of messages", "autoResize": true, "lineHeight": 1.25 }, { "id": "cOQuXVHzDRukZ8G46BB8m", "type": "text", "x": 595.0547519259976, "y": 16.272248358793604, "width": 423.59967041015625, "height": 125, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ax", "roundness": null, "seed": 1270963977, "version": 109, "versionNonce": 566168969, "isDeleted": false, "boundElements": [], "updated": 1773096957344, "link": null, "locked": false, "text": "Unique log_ids can be generated using name\nbased UUIDs using:\n- timestamp\n- application_id\n- message_hash", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Unique log_ids can be generated using name\nbased UUIDs using:\n- timestamp\n- application_id\n- message_hash", "autoResize": true, "lineHeight": 1.25 }, { "id": "Y5gTfOATaURV9q12TtdlR", "type": "text", "x": 385.0547519259976, "y": 440.2722483587936, "width": 377.91973876953125, "height": 125, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ay", "roundness": null, "seed": 1683659335, "version": 154, "versionNonce": 426902631, "isDeleted": false, "boundElements": [], "updated": 1773097443325, "link": null, "locked": false, "text": "Connection Servers maintain map\n{application_id: SSE_connection}\n\nand use this to push events to clients\nsubscribed to specific applicatoin logs", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Connection Servers maintain map\n{application_id: SSE_connection}\n\nand use this to push events to clients\nsubscribed to specific applicatoin logs", "autoResize": true, "lineHeight": 1.25 }, { "id": "EfcUAaqbwMsdF12-IHi0p", "type": "text", "x": 1371.0547519259976, "y": 166.2722483587936, "width": 214.83984375, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b00", "roundness": null, "seed": 1416228199, "version": 22, "versionNonce": 2078752519, "isDeleted": false, "boundElements": [], "updated": 1773097465686, "link": null, "locked": false, "text": "DDB stram on inserts", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "DDB stram on inserts", "autoResize": true, "lineHeight": 1.25 }, { "id": "ui9t7oomGbj1S4al_5DG_", "type": "text", "x": 1550.4766269259976, "y": -103.2590016412064, "width": 400.5797119140625, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b01", "roundness": null, "seed": 768414983, "version": 68, "versionNonce": 209623783, "isDeleted": false, "boundElements": [], "updated": 1773097498152, "link": null, "locked": false, "text": "snapshot at end of day + compress data", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "snapshot at end of day + compress data", "autoResize": true, "lineHeight": 1.25 }, { "id": "i5fAVckXcbRx7yAFLLHKw", "type": "text", "x": 1213.3828769259976, "y": -211.5402516412064, "width": 351.95977783203125, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b02", "roundness": null, "seed": 2054033479, "version": 51, "versionNonce": 1041746217, "isDeleted": false, "boundElements": [], "updated": 1773097518211, "link": null, "locked": false, "text": "DDB TTL for 7 days (hot storage)", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "DDB TTL for 7 days (hot storage)", "autoResize": true, "lineHeight": 1.25 }, { "id": "HKYSHoxHLfXzUgjGF2c2H", "type": "arrow", "x": 420.3672519259976, "y": -233.6965016412064, "width": 151.078125, "height": 87.21875, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b03", "roundness": { "type": 2 }, "seed": 1192077769, "version": 55, "versionNonce": 982929033, "isDeleted": false, "boundElements": [ { "type": "text", "id": "vvfWQo1_fM-sBPnZNTfne" } ], "updated": 1773097563084, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -151.078125, 87.21875 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "LsvHg_AckHR6fVPDm7ldK", "focus": -0.46962966308735615, "gap": 8.000868055555117 }, "endBinding": { "elementId": "TSsgrNZWM82-QJgalv47k", "focus": -0.43184262938741463, "gap": 7.247704399956547 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "vvfWQo1_fM-sBPnZNTfne", "type": "text", "x": 315.83820682101714, "y": -202.5871266412064, "width": 57.97996520996094, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b04", "roundness": null, "seed": 218601287, "version": 8, "versionNonce": 1561763911, "isDeleted": false, "boundElements": [], "updated": 1773097560890, "link": null, "locked": false, "text": "log_id", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "HKYSHoxHLfXzUgjGF2c2H", "originalText": "log_id", "autoResize": true, "lineHeight": 1.25 }, { "id": "6GwWwSBiz-MndKryG92bG", "type": "rectangle", "x": 426.6172519259976, "y": -563.2902516412064, "width": 129.53125, "height": 115.28125, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b05", "roundness": { "type": 3 }, "seed": 1102943593, "version": 52, "versionNonce": 1723805767, "isDeleted": false, "boundElements": [ { "type": "text", "id": "fLKIYziD1ohGzE8KkhY2i" }, { "id": "K_9GeP6M4vygnp2InIJZm", "type": "arrow" } ], "updated": 1773097601482, "link": null, "locked": false }, { "id": "fLKIYziD1ohGzE8KkhY2i", "type": "text", "x": 455.562900119357, "y": -555.6496266412064, "width": 71.63995361328125, "height": 100, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b06", "roundness": null, "seed": 737752841, "version": 26, "versionNonce": 1644330441, "isDeleted": false, "boundElements": [], "updated": 1773097579690, "link": null, "locked": false, "text": "Redis\n\nDe Dup\nCache", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "6GwWwSBiz-MndKryG92bG", "originalText": "Redis\n\nDe Dup\nCache", "autoResize": true, "lineHeight": 1.25 }, { "id": "1mVOYY4-WUiPD83A0FsfO", "type": "text", "x": 593.0547519259976, "y": -525.7277516412064, "width": 156.53988647460938, "height": 75, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b07", "roundness": null, "seed": 155974023, "version": 47, "versionNonce": 137827655, "isDeleted": false, "boundElements": [], "updated": 1773097596427, "link": null, "locked": false, "text": "Timestamp\n+ applicatoin_id\n+ message_hash", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Timestamp\n+ applicatoin_id\n+ message_hash", "autoResize": true, "lineHeight": 1.25 }, { "id": "K_9GeP6M4vygnp2InIJZm", "type": "arrow", "x": 461.9922519259976, "y": -350.2277516412064, "width": 27.375, "height": 91.71875, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b08", "roundness": { "type": 2 }, "seed": 384413575, "version": 39, "versionNonce": 100232487, "isDeleted": false, "boundElements": [], "updated": 1773097601482, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 27.375, -91.71875 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "LsvHg_AckHR6fVPDm7ldK", "focus": -0.10157101767800167, "gap": 4.5720486111108585 }, "endBinding": { "elementId": "6GwWwSBiz-MndKryG92bG", "focus": -0.20736560214206015, "gap": 6.0625 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "QsyA4zris0vYop3xMzCGV", "type": "text", "x": -959.4374355740024, "y": 846.1706858587936, "width": 1091.8592529296875, "height": 750, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b09", "roundness": null, "seed": 615808809, "version": 35, "versionNonce": 827620359, "isDeleted": false, "boundElements": [], "updated": 1773097849064, "link": null, "locked": false, "text": "Redis for deduplication:\nCreate key using application_id:message_hash\nSet TTL for 10 secs\nCreate redis script to check and insert in cache as a single atmoic transaction \n(this prevents multiple processes to write to the cache at once - needed for effective de deuplication).\n \n\nIngestion kafka:\nUsed to decouple ingestion and log processing stages\nNeeded to ensure low latency write to log system file\nPartiton key in kafka - application_id\nOnce message is produced to kafka, we can return log_id since kafka guarantes atleast once message delivery.\n \n\nDynamoDB for querying recent logs:\nDDB has parition key on application_id and sort_key as timestamp#log_id to get time ordered logs.\nDDB can scale horizontally and we can set replication factor to spread writes.\n \n\nES for full text search:\nWe ingest and store data in ES for supporting full text search on logs.\n \n\nLive Log tailing:\nWe use redis pub/sub and SSE events to ensure all processed logs are avialable to clients on a per app basis.\n \n\n**Kafka for decoupling live **:\nWe use kafka streaming live log tailing so that clients can replay logs\nkafka also provides durability incase redis pub/sub fails to deliver a message since it doesn't guarantee delivery", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Redis for deduplication:\nCreate key using application_id:message_hash\nSet TTL for 10 secs\nCreate redis script to check and insert in cache as a single atmoic transaction \n(this prevents multiple processes to write to the cache at once - needed for effective de deuplication).\n \n\nIngestion kafka:\nUsed to decouple ingestion and log processing stages\nNeeded to ensure low latency write to log system file\nPartiton key in kafka - application_id\nOnce message is produced to kafka, we can return log_id since kafka guarantes atleast once message delivery.\n \n\nDynamoDB for querying recent logs:\nDDB has parition key on application_id and sort_key as timestamp#log_id to get time ordered logs.\nDDB can scale horizontally and we can set replication factor to spread writes.\n \n\nES for full text search:\nWe ingest and store data in ES for supporting full text search on logs.\n \n\nLive Log tailing:\nWe use redis pub/sub and SSE events to ensure all processed logs are avialable to clients on a per app basis.\n \n\n**Kafka for decoupling live **:\nWe use kafka streaming live log tailing so that clients can replay logs\nkafka also provides durability incase redis pub/sub fails to deliver a message since it doesn't guarantee delivery", "autoResize": true, "lineHeight": 1.25 }, { "id": "D-KbXFiXD9IR-g8BaQBs9", "type": "text", "x": 621.9422783468829, "y": 840.9020958067798, "width": 1231.359130859375, "height": 325, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0B", "roundness": null, "seed": 1641924699, "version": 11, "versionNonce": 92148917, "isDeleted": false, "boundElements": null, "updated": 1773353686829, "link": null, "locked": false, "text": "These logs can be ingested using two approaches:\n\nPush Model: Logs are pushed directly to the system via agents running alongside the microservices. Example: AWS CloudWatch.\nPull Model: A central component polls endpoints exposed by the microservices to fetch logs. Example: Prometheus.\nI’d choose the pull model for greater control and scalability. A central Poller can dynamically adjust to traffic patterns and \nmanage compute resources efficiently. I’ll use ZooKeeper to maintain a registry of service endpoints for polling.\n\nTrade-offs:\n\nPush Model Pros: Simple implementation; real-time log delivery.\nPush Model Cons: Agents increase resource usage on microservices.\nPull Model Pros: Centralized control; easier scaling.\nPull Model Cons: Slightly higher latency due to polling intervals.", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "These logs can be ingested using two approaches:\n\nPush Model: Logs are pushed directly to the system via agents running alongside the microservices. Example: AWS CloudWatch.\nPull Model: A central component polls endpoints exposed by the microservices to fetch logs. Example: Prometheus.\nI’d choose the pull model for greater control and scalability. A central Poller can dynamically adjust to traffic patterns and \nmanage compute resources efficiently. I’ll use ZooKeeper to maintain a registry of service endpoints for polling.\n\nTrade-offs:\n\nPush Model Pros: Simple implementation; real-time log delivery.\nPush Model Cons: Agents increase resource usage on microservices.\nPull Model Pros: Centralized control; easier scaling.\nPull Model Cons: Slightly higher latency due to polling intervals.", "autoResize": true, "lineHeight": 1.25 }, { "id": "bhmFUarlmBAxdPfNze4K0", "type": "text", "x": 612.9774345968829, "y": 1281.41967393178, "width": 1479.216279315947, "height": 114.6093749999999, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0C", "roundness": null, "seed": 92513909, "version": 85, "versionNonce": 1329799829, "isDeleted": false, "boundElements": null, "updated": 1773354248515, "link": null, "locked": false, "text": "Long-term storage, I’ll introduce a Storage Service that consumes logs from Kafka and writes them to a multi-tier storage solution:\n\nHot Storage: Recent logs are stored in S3 in an efficient format like Apache Parquet for analytical queries.\nCold Storage: After 30 days, logs are moved to a cheaper storage tier like AWS Glacier for archival purposes.", "fontSize": 22.921874999999982, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Long-term storage, I’ll introduce a Storage Service that consumes logs from Kafka and writes them to a multi-tier storage solution:\n\nHot Storage: Recent logs are stored in S3 in an efficient format like Apache Parquet for analytical queries.\nCold Storage: After 30 days, logs are moved to a cheaper storage tier like AWS Glacier for archival purposes.", "autoResize": true, "lineHeight": 1.25 }, { "id": "WLyLBsedQGXO_RiV11L1-", "type": "text", "x": 627.9774345968829, "y": 1461.41967393178, "width": 976.9193725585938, "height": 325, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0D", "roundness": null, "seed": 1998309307, "version": 3, "versionNonce": 66278651, "isDeleted": false, "boundElements": null, "updated": 1773354268386, "link": null, "locked": false, "text": "The Storage Service will enforce compliance policies (e.g., GDPR) to manage retention and deletions.\n\nInterviewer: Why S3 and Parquet?\n\nInterviewee:\n\nS3 Pros: Highly available, cost-effective, and scales effortlessly.\nParquet Pros: Columnar format reduces storage costs and accelerates queries.\nCold Storage: Glacier minimizes costs for rarely accessed data.\nTrade-offs:\n\nHot Storage Cons: Slightly higher costs for recent data.\nCold Storage Cons: Higher retrieval latency.", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "The Storage Service will enforce compliance policies (e.g., GDPR) to manage retention and deletions.\n\nInterviewer: Why S3 and Parquet?\n\nInterviewee:\n\nS3 Pros: Highly available, cost-effective, and scales effortlessly.\nParquet Pros: Columnar format reduces storage costs and accelerates queries.\nCold Storage: Glacier minimizes costs for rarely accessed data.\nTrade-offs:\n\nHot Storage Cons: Slightly higher costs for recent data.\nCold Storage Cons: Higher retrieval latency.", "autoResize": true, "lineHeight": 1.25 } ], "appState": { "gridSize": 20, "gridStep": 5, "gridModeEnabled": false, "viewBackgroundColor": "#ffffff" }, "files": {} }